2024-03-26T03:57:13.135 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T03:57:13.136 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985 --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985/orig.config.yaml 2024-03-26T03:57:13.162 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T03:57:13.246 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7621985' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPHjTUqpJSdOkZX47TJiip8kRwGlE9H0aSdHyJDR4DlNZh3CHcg0TlTvv/X7xY/WwlAKTslNdktBxRX5pVtKSCg= smithi149.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2FUuQaQr44IF38rP8jG19B6CXq+9A3huVRWHjs/gnGGNx6xov3bC7mcPK+S/ZqWV9pMfRBRAAn83lYFe1t0ng= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRA5xmGNfBUGCo9IzqoIoh55PFwHg9rn7/uVloMm9FqMjywdxIhxjS9jV//qAdSyhfTYm9UDMGjNmavbEeaCiA= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T03:57:13.246 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T03:57:13.247 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T03:57:13.247 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T03:57:13.249 INFO:teuthology.task.internal:Checking packages... 2024-03-26T03:57:13.271 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T03:57:13.271 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T03:57:13.271 INFO:teuthology.packaging:ref: None 2024-03-26T03:57:13.272 INFO:teuthology.packaging:tag: None 2024-03-26T03:57:13.272 INFO:teuthology.packaging:branch: squid 2024-03-26T03:57:13.272 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T03:57:13.272 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T03:57:13.427 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T03:57:13.428 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T03:57:13.434 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T03:57:13.434 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T03:57:13.479 INFO:teuthology.task.internal:Saving configuration 2024-03-26T03:57:13.490 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T03:57:13.513 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T03:57:13.537 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:51:33.179160', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPHjTUqpJSdOkZX47TJiip8kRwGlE9H0aSdHyJDR4DlNZh3CHcg0TlTvv/X7xY/WwlAKTslNdktBxRX5pVtKSCg='} 2024-03-26T03:57:13.560 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:51:33.178306', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2FUuQaQr44IF38rP8jG19B6CXq+9A3huVRWHjs/gnGGNx6xov3bC7mcPK+S/ZqWV9pMfRBRAAn83lYFe1t0ng='} 2024-03-26T03:57:13.580 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:51:33.177273', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRA5xmGNfBUGCo9IzqoIoh55PFwHg9rn7/uVloMm9FqMjywdxIhxjS9jV//qAdSyhfTYm9UDMGjNmavbEeaCiA='} 2024-03-26T03:57:13.581 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T03:57:13.586 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T03:57:13.586 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-26T03:57:13.587 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-26T03:57:13.587 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T03:57:13.733 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb918d53f70>, signals=[15]) 2024-03-26T03:57:13.734 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T03:57:13.740 INFO:teuthology.task.internal:Opening connections... 2024-03-26T03:57:13.740 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-03-26T03:57:13.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:57:13.835 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2024-03-26T03:57:13.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:57:13.926 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-03-26T03:57:13.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:57:14.015 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T03:57:14.022 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-03-26T03:57:14.039 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-03-26T03:57:14.040 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-03-26T03:57:14.096 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-03-26T03:57:14.096 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-03-26T03:57:14.096 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-03-26T03:57:14.096 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-03-26T03:57:14.096 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-03-26T03:57:14.097 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-03-26T03:57:14.097 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T03:57:14.097 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-03-26T03:57:14.097 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-03-26T03:57:14.097 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T03:57:14.097 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-03-26T03:57:14.097 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T03:57:14.097 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T03:57:14.097 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T03:57:14.098 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-03-26T03:57:14.126 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2024-03-26T03:57:14.143 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2024-03-26T03:57:14.144 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Stream" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="9" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="9" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el9" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:LOGO="fedora-logo-icon" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T03:57:14.201 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T03:57:14.202 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2024-03-26T03:57:14.226 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-03-26T03:57:14.244 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-03-26T03:57:14.244 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-03-26T03:57:14.301 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-03-26T03:57:14.301 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-03-26T03:57:14.301 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-03-26T03:57:14.301 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-03-26T03:57:14.301 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-03-26T03:57:14.301 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-03-26T03:57:14.301 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T03:57:14.301 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-03-26T03:57:14.302 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-03-26T03:57:14.302 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T03:57:14.302 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-03-26T03:57:14.302 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T03:57:14.302 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T03:57:14.302 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T03:57:14.302 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-03-26T03:57:14.326 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T03:57:14.335 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T03:57:14.371 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T03:57:14.371 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-03-26T03:57:14.374 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2024-03-26T03:57:14.376 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-03-26T03:57:14.391 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T03:57:14.431 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T03:57:14.432 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-03-26T03:57:14.434 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2024-03-26T03:57:14.436 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-03-26T03:57:14.679 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T03:57:14.752 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T03:57:14.766 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T03:57:14.766 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T03:57:14.767 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T03:57:14.767 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T03:57:14.767 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T03:57:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:14.784 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-03-26T03:57:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:14.786 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-26T03:57:14.786 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-26T03:57:14.842 INFO:teuthology.orchestra.run.smithi149.stdout:5.14.0-361.el9.x86_64 2024-03-26T03:57:14.842 INFO:teuthology.task.kernel:Running kernel on smithi149: 5.14.0-361.el9.x86_64 2024-03-26T03:57:14.842 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum install -y kernel 2024-03-26T03:57:14.844 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-361.el9.x86_64 2024-03-26T03:57:14.844 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-361.el9.x86_64 2024-03-26T03:57:14.844 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.14.0-361.el9.x86_64 2024-03-26T03:57:14.845 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2024-03-26T03:57:14.845 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-361.el9.x86_64 2024-03-26T03:57:14.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-03-26T03:57:16.921 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-26T03:57:17.053 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T03:57:17.069 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-26T03:57:18.470 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-26T03:57:18.530 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-26T03:57:18.686 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-26T03:57:19.568 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-26T03:57:19.688 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-26T03:57:22.410 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-26T03:57:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 19 MB 00:02 2024-03-26T03:57:32.220 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-26T03:57:33.712 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-26T03:57:34.556 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 28 MB/s | 21 MB 00:00 2024-03-26T03:57:35.317 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-26T03:57:36.612 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-26T03:57:39.259 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-26T03:57:39.506 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-26T03:57:40.852 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-26T03:57:41.662 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T03:57:41.663 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T03:57:41.743 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-26T03:57:41.753 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-26T03:57:41.753 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-26T03:57:41.753 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-26T03:57:41.753 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-26T03:57:41.753 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T03:57:41.753 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-26T03:57:41.753 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T03:57:41.753 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T03:57:41.753 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T03:57:41.753 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T03:57:41.754 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-26T03:57:41.754 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-26T03:57:41.754 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-03-26T03:57:41.754 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T03:57:41.754 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 76 M 2024-03-26T03:57:41.754 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 123 M 2024-03-26T03:57:41.755 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-26T03:57:41.820 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-26T03:57:42.199 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 217 kB/s | 66 kB 00:00 2024-03-26T03:57:43.314 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:03 2024-03-26T03:57:45.563 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-26T03:57:46.752 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-03-26T03:57:46.782 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.1 MB/s | 15 MB 00:04 2024-03-26T03:57:46.900 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-26T03:57:47.782 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 5.7 MB/s | 33 MB 00:05 2024-03-26T03:57:48.049 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 4.7 MB/s | 27 MB 00:05 2024-03-26T03:57:48.050 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:57:48.050 INFO:teuthology.orchestra.run.smithi204.stdout:Total 12 MB/s | 76 MB 00:06 2024-03-26T03:57:48.720 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-26T03:57:48.884 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T03:57:48.884 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 76 M 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 123 M 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T03:57:49.100 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-26T03:57:49.100 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-26T03:57:49.322 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:57:49.322 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:57:49.323 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:57:49.463 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:57:49.472 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:57:49.497 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:57:49.545 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 230 kB/s | 66 kB 00:00 2024-03-26T03:57:49.578 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:57:49.612 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:57:50.130 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-26T03:57:50.130 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-26T03:57:51.696 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): kernel-modules-core-5.14.0-432.el9.x86_6 13 MB/s | 27 MB 00:02 2024-03-26T03:57:51.855 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-26T03:57:52.129 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-03-26T03:57:52.504 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): kernel-core-5.14.0-432.el9.x86_64.rpm 4.6 MB/s | 15 MB 00:03 2024-03-26T03:57:52.507 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:57:52.507 INFO:teuthology.orchestra.run.smithi136.stdout:Total 22 MB/s | 76 MB 00:03 2024-03-26T03:57:52.603 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-26T03:57:53.173 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T03:57:53.250 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:57:53.333 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:57:53.535 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T03:57:53.535 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T03:57:54.014 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T03:57:54.014 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T03:57:54.099 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T03:57:54.106 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-03-26T03:57:54.107 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-03-26T03:57:54.107 INFO:teuthology.orchestra.run.smithi149.stdout:Install 4 Packages 2024-03-26T03:57:54.107 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T03:57:54.107 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 76 M 2024-03-26T03:57:54.107 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 123 M 2024-03-26T03:57:54.107 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-03-26T03:57:54.449 INFO:teuthology.orchestra.run.smithi149.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 965 kB/s | 66 kB 00:00 2024-03-26T03:57:54.524 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T03:57:54.525 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T03:57:55.374 INFO:teuthology.orchestra.run.smithi149.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 15 MB/s | 15 MB 00:00 2024-03-26T03:57:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:57:56.116 INFO:teuthology.orchestra.run.smithi149.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 16 MB/s | 27 MB 00:01 2024-03-26T03:57:56.197 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:57:56.408 INFO:teuthology.orchestra.run.smithi149.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 17 MB/s | 33 MB 00:02 2024-03-26T03:57:56.408 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:57:56.408 INFO:teuthology.orchestra.run.smithi149.stdout:Total 33 MB/s | 76 MB 00:02 2024-03-26T03:57:56.990 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T03:57:57.078 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-03-26T03:57:57.422 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-03-26T03:57:57.422 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-03-26T03:57:57.597 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:57:57.719 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:57:58.451 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-03-26T03:57:58.451 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-03-26T03:58:00.469 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:58:00.603 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:58:00.890 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:58:00.934 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-03-26T03:58:01.280 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:01.553 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:58:01.649 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:58:04.406 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:58:04.544 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:58:05.275 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:58:05.395 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:06.010 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:09.285 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:58:09.326 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:14.086 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:35.264 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:35.264 INFO:teuthology.orchestra.run.smithi204.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T03:58:35.264 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T03:58:35.308 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:35.672 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:35.673 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:58:35.673 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:58:35.673 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:58:36.085 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:36.085 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T03:58:36.085 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-26T03:58:36.085 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T03:58:36.085 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T03:58:36.085 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T03:58:36.086 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T03:58:36.086 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T03:58:36.086 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-26T03:58:36.415 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:58:36.508 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:58:36.508 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:58:36.508 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T03:58:36.509 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T03:58:36.509 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T03:58:36.509 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-26T03:58:36.509 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T03:58:36.509 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:58:36.589 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:58:36.589 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:58:36.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-03-26T03:58:36.624 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-03-26T03:58:36.624 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-03-26T03:58:36.624 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.06 2024-03-26T03:58:36.624 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 68.el9 2024-03-26T03:58:36.624 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-03-26T03:58:36.624 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T03:58:36.624 INFO:teuthology.orchestra.run.smithi204.stdout:Group : Unspecified 2024-03-26T03:58:36.624 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 8264254 2024-03-26T03:58:36.624 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : builder@centos.org 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T03:58:36.625 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-03-26T03:58:36.626 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T03:58:36.626 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T03:58:36.626 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T03:58:36.627 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 5.14.0-432.el9.x86_64 2024-03-26T03:58:36.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T03:58:37.211 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-03-26T03:58:38.596 INFO:teuthology.orchestra.run.smithi204.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T03:58:38.635 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-03-26T03:58:38.637 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-03-26T03:58:38.652 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.Yy7CUUL0C0 2024-03-26T03:58:38.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Yy7CUUL0C0 2024-03-26T03:58:38.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.Yy7CUUL0C0 2024-03-26T03:58:38.857 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.Yy7CUUL0C0 is 6KB 2024-03-26T03:58:38.877 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.Yy7CUUL0C0 2024-03-26T03:58:38.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T03:58:38.962 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T03:58:38.962 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T03:58:38.962 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T03:58:38.962 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T03:58:38.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T03:58:39.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-03-26T03:58:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:39.504 INFO:teuthology.orchestra.run.smithi136.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T03:58:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T03:58:39.549 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:39.972 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:39.972 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:58:39.972 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:58:39.973 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:58:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T03:58:40.412 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T03:58:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T03:58:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T03:58:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T03:58:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T03:58:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T03:58:40.413 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T03:58:40.785 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:58:40.880 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:58:40.880 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:58:40.880 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T03:58:40.880 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T03:58:40.880 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T03:58:40.881 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T03:58:40.881 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T03:58:40.881 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:58:40.960 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:58:40.961 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:58:40.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo rpm -qi grub2-tools 2024-03-26T03:58:40.994 INFO:teuthology.orchestra.run.smithi136.stdout:Name : grub2-tools 2024-03-26T03:58:40.994 INFO:teuthology.orchestra.run.smithi136.stdout:Epoch : 1 2024-03-26T03:58:40.994 INFO:teuthology.orchestra.run.smithi136.stdout:Version : 2.06 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Release : 68.el9 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: x86_64 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Group : Unspecified 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Size : 8264254 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:License : GPLv3+ 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Packager : builder@centos.org 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Vendor : CentOS 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Summary : Support tools for GRUB. 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout:Description : 2024-03-26T03:58:40.995 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T03:58:40.996 INFO:teuthology.orchestra.run.smithi136.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T03:58:40.996 INFO:teuthology.orchestra.run.smithi136.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T03:58:40.996 INFO:teuthology.orchestra.run.smithi136.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T03:58:40.996 INFO:teuthology.orchestra.run.smithi136.stdout:hardware devices. 2024-03-26T03:58:40.996 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T03:58:40.996 INFO:teuthology.orchestra.run.smithi136.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T03:58:40.997 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T03:58:40.997 INFO:teuthology.task.kernel:Updating grub on smithi136 to boot 5.14.0-432.el9.x86_64 2024-03-26T03:58:40.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T03:58:41.643 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-03-26T03:58:43.060 INFO:teuthology.orchestra.run.smithi136.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T03:58:43.094 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-03-26T03:58:43.097 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-03-26T03:58:43.152 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.UQy9zKHGJS 2024-03-26T03:58:43.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UQy9zKHGJS 2024-03-26T03:58:43.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.UQy9zKHGJS 2024-03-26T03:58:43.360 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.UQy9zKHGJS is 6KB 2024-03-26T03:58:43.381 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:43.381 INFO:teuthology.orchestra.run.smithi149.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T03:58:43.381 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T03:58:43.408 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.UQy9zKHGJS 2024-03-26T03:58:43.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T03:58:43.426 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:43.488 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T03:58:43.488 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T03:58:43.488 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T03:58:43.488 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T03:58:43.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T03:58:43.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-03-26T03:58:43.823 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:43.823 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:58:43.823 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:58:43.823 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:58:44.251 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:58:44.252 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T03:58:44.252 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-03-26T03:58:44.252 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T03:58:44.252 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T03:58:44.252 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T03:58:44.252 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T03:58:44.252 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T03:58:44.252 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-03-26T03:58:44.552 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:58:44.695 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:58:44.696 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:58:44.696 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T03:58:44.696 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T03:58:44.696 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T03:58:44.696 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T03:58:44.696 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi149.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T03:58:44.696 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:58:44.777 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:58:44.777 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:58:44.777 DEBUG:teuthology.orchestra.run.smithi149:> sudo rpm -qi grub2-tools 2024-03-26T03:58:44.811 INFO:teuthology.orchestra.run.smithi149.stdout:Name : grub2-tools 2024-03-26T03:58:44.811 INFO:teuthology.orchestra.run.smithi149.stdout:Epoch : 1 2024-03-26T03:58:44.811 INFO:teuthology.orchestra.run.smithi149.stdout:Version : 2.06 2024-03-26T03:58:44.811 INFO:teuthology.orchestra.run.smithi149.stdout:Release : 68.el9 2024-03-26T03:58:44.811 INFO:teuthology.orchestra.run.smithi149.stdout:Architecture: x86_64 2024-03-26T03:58:44.811 INFO:teuthology.orchestra.run.smithi149.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T03:58:44.811 INFO:teuthology.orchestra.run.smithi149.stdout:Group : Unspecified 2024-03-26T03:58:44.811 INFO:teuthology.orchestra.run.smithi149.stdout:Size : 8264254 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:License : GPLv3+ 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:Packager : builder@centos.org 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:Vendor : CentOS 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:Summary : Support tools for GRUB. 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:Description : 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T03:58:44.812 INFO:teuthology.orchestra.run.smithi149.stdout:hardware devices. 2024-03-26T03:58:44.813 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T03:58:44.813 INFO:teuthology.orchestra.run.smithi149.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T03:58:44.813 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T03:58:44.814 INFO:teuthology.task.kernel:Updating grub on smithi149 to boot 5.14.0-432.el9.x86_64 2024-03-26T03:58:44.814 DEBUG:teuthology.orchestra.run.smithi149:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T03:58:45.466 INFO:teuthology.orchestra.run.smithi149.stderr:Generating grub configuration file ... 2024-03-26T03:58:46.919 INFO:teuthology.orchestra.run.smithi149.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T03:58:46.954 INFO:teuthology.orchestra.run.smithi149.stderr:done 2024-03-26T03:58:46.963 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2024-03-26T03:58:46.988 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.K4wXcjNELo 2024-03-26T03:58:46.988 DEBUG:teuthology.orchestra.run.smithi149:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.K4wXcjNELo 2024-03-26T03:58:47.057 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.K4wXcjNELo 2024-03-26T03:58:47.196 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.K4wXcjNELo is 6KB 2024-03-26T03:58:47.245 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.K4wXcjNELo 2024-03-26T03:58:47.259 DEBUG:teuthology.orchestra.run.smithi149:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T03:58:47.324 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T03:58:47.325 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T03:58:47.325 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T03:58:47.325 INFO:teuthology.orchestra.run.smithi149.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T03:58:47.326 DEBUG:teuthology.orchestra.run.smithi149:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T03:58:47.474 DEBUG:teuthology.orchestra.run.smithi149:> sudo shutdown -r now 2024-03-26T03:59:09.126 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-26T03:59:09.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-26T03:59:09.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:59:13.629 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T03:59:13.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T03:59:13.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:59:17.478 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T03:59:17.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-26T03:59:17.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:59:27.519 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-26T03:59:32.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-26T03:59:35.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-03-26T03:59:36.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-26T03:59:36.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:59:39.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-26T03:59:41.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T03:59:41.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:59:44.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-26T03:59:44.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-26T03:59:44.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:59:48.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-03-26T03:59:51.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-26T03:59:51.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:59:56.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T03:59:56.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:59:59.266 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-26T04:00:00.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-26T04:00:00.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:00:10.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-26T04:00:14.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T04:00:14.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:00:17.346 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-26T04:00:25.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-26T04:00:25.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:00:28.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-26T04:00:35.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T04:00:35.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:00:35.640 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-26T04:00:36.024 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T04:00:36.024 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:00:36.024 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-26T04:00:36.040 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:00:36.040 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:00:36.040 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:00:36.040 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:00:37.041 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T04:00:37.041 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:00:37.117 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-03-26T04:00:37.153 DEBUG:teuthology.parallel:result is None 2024-03-26T04:00:46.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-26T04:00:46.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:00:46.564 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-26T04:00:46.959 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-26T04:00:46.960 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:00:46.960 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-26T04:00:46.975 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:00:46.976 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:00:46.976 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:00:46.976 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:00:47.977 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-26T04:00:47.977 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:00:48.054 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-03-26T04:00:48.081 DEBUG:teuthology.parallel:result is None 2024-03-26T04:01:00.034 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T04:01:15.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-26T04:01:15.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:01:15.394 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-03-26T04:01:15.790 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-26T04:01:15.790 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:01:15.790 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-03-26T04:01:15.805 INFO:teuthology.orchestra.run.smithi149.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:01:15.805 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:01:15.805 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:01:15.805 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:01:16.806 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T04:01:16.806 DEBUG:teuthology.orchestra.run.smithi149:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:01:16.875 INFO:teuthology.orchestra.run.smithi149.stdout:ttyS1 2024-03-26T04:01:16.917 DEBUG:teuthology.parallel:result is None 2024-03-26T04:01:16.918 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T04:01:16.925 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T04:01:16.925 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:01:16.928 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:01:16.930 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:01:16.951 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T04:01:16.967 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T04:01:17.018 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T04:01:17.018 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:01:17.021 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:01:17.023 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:01:17.074 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T04:01:17.105 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T04:01:17.106 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:01:17.110 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:01:17.112 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:01:17.142 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:01:17.147 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:01:17.156 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:01:17.156 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:01:17.161 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:01:17.171 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:01:17.172 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T04:01:17.180 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T04:01:17.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:01:17.206 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:01:17.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:01:17.237 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T04:01:17.247 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T04:01:17.247 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:01:17.277 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:01:17.279 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:01:17.295 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:01:17.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:01:17.422 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:01:17.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:01:17.548 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:01:17.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:01:17.615 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:01:17.659 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:01:17.711 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:01:17.795 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:01:17.841 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:01:17.842 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:01:17.910 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:01:17.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:01:18.009 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:01:18.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:01:18.138 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:01:18.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:01:18.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-03-26T04:01:18.211 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2024-03-26T04:01:18.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-03-26T04:01:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:01:18.254 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:01:18.291 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:01:18.781 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T04:01:18.790 INFO:teuthology.task.internal:Starting timer... 2024-03-26T04:01:18.790 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T04:01:18.840 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T04:01:18.945 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T04:01:18.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-03-26T04:01:18.996 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-03-26T04:01:18.998 DEBUG:teuthology.orchestra.run.smithi149:> sudo service auditd rotate 2024-03-26T04:01:19.046 INFO:teuthology.orchestra.run.smithi149.stdout:Rotating logs: 2024-03-26T04:01:19.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-03-26T04:01:19.105 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-03-26T04:01:19.107 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T04:01:19.107 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-03-26T04:01:19.129 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-03-26T04:01:19.129 DEBUG:teuthology.orchestra.run.smithi149:> /usr/sbin/getenforce 2024-03-26T04:01:19.155 INFO:teuthology.orchestra.run.smithi149.stdout:Permissive 2024-03-26T04:01:19.155 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-03-26T04:01:19.177 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-03-26T04:01:19.177 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi136.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi149.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-03-26T04:01:19.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:01:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:19.207 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:01:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:19.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:01:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:19.266 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T04:01:19.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:01:19.306 DEBUG:teuthology.orchestra.run.smithi149:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:01:19.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:01:19.378 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T04:01:19.389 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T04:01:19.409 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T04:01:19.411 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi136.front.sepia.ceph.com,smithi149.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T04:05:50.375 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi136.front.sepia.ceph.com'), Remote(name='ubuntu@smithi149.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-03-26T04:05:50.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T04:05:50.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:05:50.471 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-26T04:05:50.550 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T04:05:50.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-26T04:05:50.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:05:50.638 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-03-26T04:05:50.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-03-26T04:05:50.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-26T04:05:50.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:05:50.805 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-26T04:05:50.887 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-26T04:05:50.887 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T04:05:50.897 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T04:05:50.898 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:05:50.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:05:50.901 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:05:50.901 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:05:50.903 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:05:50.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:05:50.933 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:05:50.934 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:05:50.951 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:05:50.952 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:05:50.959 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:05:50.976 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:05:50.981 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: ntpd: command not found 2024-03-26T04:05:50.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-03-26T04:05:50.998 INFO:teuthology.orchestra.run.smithi149.stdout:506 Cannot talk to daemon 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-03-26T04:05:51.006 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-03-26T04:05:51.012 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:05:51.015 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:05:51.020 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-03-26T04:05:51.029 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:05:51.031 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:05:51.036 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:05:51.053 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:05:51.079 INFO:teuthology.orchestra.run.smithi149.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:05:51.082 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:05:51.082 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-03-26T04:05:51.082 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.083 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.083 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.083 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.084 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:05:51.087 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:05:51.087 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-26T04:05:51.087 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.087 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.087 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.088 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.103 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:05:51.106 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:05:51.106 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-26T04:05:51.106 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.106 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.106 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.106 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:05:51.107 INFO:teuthology.run_tasks:Running task pexec... 2024-03-26T04:05:51.116 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-26T04:05:51.116 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi136.front.sepia.ceph.com 2024-03-26T04:05:51.117 DEBUG:teuthology.orchestra.run.smithi136:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T04:05:51.117 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi149.front.sepia.ceph.com 2024-03-26T04:05:51.117 DEBUG:teuthology.orchestra.run.smithi149:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T04:05:51.117 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi204.front.sepia.ceph.com 2024-03-26T04:05:51.117 DEBUG:teuthology.orchestra.run.smithi204:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T04:05:51.703 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:03 ago on Tue 26 Mar 2024 04:04:48 AM UTC. 2024-03-26T04:05:51.720 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:04 ago on Tue 26 Mar 2024 04:04:47 AM UTC. 2024-03-26T04:05:51.729 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:02 ago on Tue 26 Mar 2024 04:04:49 AM UTC. 2024-03-26T04:05:51.848 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T04:05:51.850 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T04:05:51.850 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-26T04:05:51.850 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T04:05:51.850 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T04:05:51.850 INFO:teuthology.orchestra.run.smithi136.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T04:05:51.850 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:05:51.850 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T04:05:51.850 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T04:05:51.850 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-26T04:05:51.850 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:05:51.851 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 3.1 M 2024-03-26T04:05:51.851 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 9.8 M 2024-03-26T04:05:51.851 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T04:05:51.869 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-03-26T04:05:51.870 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-03-26T04:05:51.870 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-03-26T04:05:51.870 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-03-26T04:05:51.870 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-03-26T04:05:51.870 INFO:teuthology.orchestra.run.smithi149.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T04:05:51.871 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T04:05:51.871 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-03-26T04:05:51.871 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-03-26T04:05:51.871 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-03-26T04:05:51.871 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T04:05:51.871 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 3.1 M 2024-03-26T04:05:51.871 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 9.8 M 2024-03-26T04:05:51.872 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-03-26T04:05:51.878 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-26T04:05:51.886 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-26T04:05:51.886 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 3.1 M 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 9.8 M 2024-03-26T04:05:51.887 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-26T04:05:52.167 INFO:teuthology.orchestra.run.smithi136.stdout:runc-1.1.12-2.el9.x86_64.rpm 13 MB/s | 3.1 MB 00:00 2024-03-26T04:05:52.167 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:05:52.167 INFO:teuthology.orchestra.run.smithi136.stdout:Total 9.9 MB/s | 3.1 MB 00:00 2024-03-26T04:05:52.174 INFO:teuthology.orchestra.run.smithi149.stdout:runc-1.1.12-2.el9.x86_64.rpm 13 MB/s | 3.1 MB 00:00 2024-03-26T04:05:52.175 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:05:52.175 INFO:teuthology.orchestra.run.smithi149.stdout:Total 10 MB/s | 3.1 MB 00:00 2024-03-26T04:05:52.203 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T04:05:52.208 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T04:05:52.208 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T04:05:52.211 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-03-26T04:05:52.217 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-03-26T04:05:52.217 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-03-26T04:05:52.287 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T04:05:52.287 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T04:05:52.297 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-03-26T04:05:52.298 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-03-26T04:05:52.503 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T04:05:52.526 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-03-26T04:05:52.603 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:05:52.619 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:05:53.069 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:05:53.186 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:05:53.280 INFO:teuthology.orchestra.run.smithi204.stdout:runc-1.1.12-2.el9.x86_64.rpm 2.8 MB/s | 3.1 MB 00:01 2024-03-26T04:05:53.280 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:05:53.280 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.2 MB/s | 3.1 MB 00:01 2024-03-26T04:05:53.318 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-26T04:05:53.324 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-26T04:05:53.324 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-26T04:05:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:05:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:05:53.358 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T04:05:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T04:05:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:05:53.358 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T04:05:53.407 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-26T04:05:53.407 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-26T04:05:53.466 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:05:53.467 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T04:05:53.467 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-03-26T04:05:53.467 INFO:teuthology.orchestra.run.smithi149.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T04:05:53.467 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-03-26T04:05:53.467 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-03-26T04:05:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-26T04:05:53.638 DEBUG:teuthology.parallel:result is None 2024-03-26T04:05:53.706 DEBUG:teuthology.parallel:result is None 2024-03-26T04:05:53.707 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:05:54.190 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:05:54.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T04:05:54.488 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T04:05:54.488 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-26T04:05:54.488 INFO:teuthology.orchestra.run.smithi204.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T04:05:54.488 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T04:05:54.488 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-26T04:05:54.743 DEBUG:teuthology.parallel:result is None 2024-03-26T04:05:54.743 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-26T04:05:54.754 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-26T04:05:54.754 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:05:54.754 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:05:54.773 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:05:54.774 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-03-26T04:05:54.831 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:05:54.831 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:54.831 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-26T04:05:54.831 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:54.831 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:54.831 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 04:05:49.340265619 +0000 2024-03-26T04:05:54.831 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 04:05:13.504068250 +0000 2024-03-26T04:05:54.831 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 04:05:13.504068250 +0000 2024-03-26T04:05:54.831 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 04:05:13.504068250 +0000 2024-03-26T04:05:54.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:05:54.899 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T04:05:54.899 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T04:05:54.899 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000218477 s, 2.3 MB/s 2024-03-26T04:05:54.901 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:05:54.958 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-03-26T04:05:55.015 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:05:55.015 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:55.015 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-26T04:05:55.015 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:55.016 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:55.016 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 04:05:49.340265619 +0000 2024-03-26T04:05:55.016 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 04:05:13.838060770 +0000 2024-03-26T04:05:55.016 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 04:05:13.838060770 +0000 2024-03-26T04:05:55.016 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 04:05:13.838060770 +0000 2024-03-26T04:05:55.016 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:05:55.082 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T04:05:55.082 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T04:05:55.082 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000179388 s, 2.9 MB/s 2024-03-26T04:05:55.083 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:05:55.140 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-03-26T04:05:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:05:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:55.195 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-26T04:05:55.195 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:55.195 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:55.195 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 04:05:49.340265619 +0000 2024-03-26T04:05:55.195 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 04:05:14.179053132 +0000 2024-03-26T04:05:55.196 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 04:05:14.179053132 +0000 2024-03-26T04:05:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 04:05:14.179053132 +0000 2024-03-26T04:05:55.196 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:05:55.261 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T04:05:55.261 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T04:05:55.261 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000174206 s, 2.9 MB/s 2024-03-26T04:05:55.262 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:05:55.318 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-03-26T04:05:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:05:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:55.374 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T04:05:55.374 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:55.374 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:55.374 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 04:05:49.340265619 +0000 2024-03-26T04:05:55.374 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 04:05:14.513045652 +0000 2024-03-26T04:05:55.374 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 04:05:14.513045652 +0000 2024-03-26T04:05:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 04:05:14.513045652 +0000 2024-03-26T04:05:55.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:05:55.440 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T04:05:55.440 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T04:05:55.440 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000227826 s, 2.2 MB/s 2024-03-26T04:05:55.441 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:05:55.496 DEBUG:teuthology.orchestra.run.smithi136:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T04:05:55.640 INFO:teuthology.orchestra.run.smithi136.stdout:loop 2024-03-26T04:05:55.642 INFO:tasks.nvme_loop:Connecting nvme_loop smithi136:/dev/vg_nvme/lv_1... 2024-03-26T04:05:55.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T04:05:55.679 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-03-26T04:05:55.716 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/vg_nvme/lv_11 2024-03-26T04:05:55.753 INFO:tasks.nvme_loop:Connecting nvme_loop smithi136:/dev/vg_nvme/lv_2... 2024-03-26T04:05:55.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T04:05:55.833 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-03-26T04:05:55.872 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/vg_nvme/lv_21 2024-03-26T04:05:55.901 INFO:tasks.nvme_loop:Connecting nvme_loop smithi136:/dev/vg_nvme/lv_3... 2024-03-26T04:05:55.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T04:05:55.980 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-03-26T04:05:56.019 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/vg_nvme/lv_31 2024-03-26T04:05:56.055 INFO:tasks.nvme_loop:Connecting nvme_loop smithi136:/dev/vg_nvme/lv_4... 2024-03-26T04:05:56.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T04:05:56.134 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-03-26T04:05:56.175 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/vg_nvme/lv_41 2024-03-26T04:05:56.204 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:05:56.204 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:05:56.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo nvme list 2024-03-26T04:05:56.325 INFO:teuthology.orchestra.run.smithi136.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T04:05:56.325 INFO:teuthology.orchestra.run.smithi136.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T04:05:56.326 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme0n1 CVPD514100122P0SGN INTEL SSDPEDMX020T4 1 2.00 TB / 2.00 TB 512 B + 0 B 8DV10171 2024-03-26T04:05:56.326 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme1n1 c55a52c167e11ef647bb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:56.326 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme2n1 26648050adf6e08f910b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:56.326 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme3n1 912fe40b46b1581d44cc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:56.326 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme4n1 90199d68f5639f4a3d36 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:56.327 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:05:56.327 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:05:56.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/scratch_devs 2024-03-26T04:05:56.394 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:05:56.394 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:05:56.410 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:05:56.410 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_1 2024-03-26T04:05:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:05:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:56.465 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-26T04:05:56.465 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:56.465 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:56.465 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-26 04:05:49.357324235 +0000 2024-03-26T04:05:56.465 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-26 04:05:14.391997398 +0000 2024-03-26T04:05:56.465 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-26 04:05:14.391997398 +0000 2024-03-26T04:05:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-26 04:05:14.391997398 +0000 2024-03-26T04:05:56.465 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:05:56.532 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-26T04:05:56.532 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-26T04:05:56.532 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000236452 s, 2.2 MB/s 2024-03-26T04:05:56.533 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:05:56.590 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_2 2024-03-26T04:05:56.646 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:05:56.646 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:56.646 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-26T04:05:56.646 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:56.646 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:56.646 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-26 04:05:49.357324235 +0000 2024-03-26T04:05:56.646 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-26 04:05:14.745990583 +0000 2024-03-26T04:05:56.646 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-26 04:05:14.745990583 +0000 2024-03-26T04:05:56.646 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-26 04:05:14.745990583 +0000 2024-03-26T04:05:56.647 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:05:56.712 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-26T04:05:56.712 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-26T04:05:56.712 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000174973 s, 2.9 MB/s 2024-03-26T04:05:56.713 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:05:56.769 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_3 2024-03-26T04:05:56.823 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:05:56.824 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:56.824 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-26T04:05:56.824 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:56.824 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:56.824 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-26 04:05:49.357324235 +0000 2024-03-26T04:05:56.824 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-26 04:05:15.091983921 +0000 2024-03-26T04:05:56.824 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-26 04:05:15.091983921 +0000 2024-03-26T04:05:56.824 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-26 04:05:15.091983921 +0000 2024-03-26T04:05:56.824 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:05:56.890 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-26T04:05:56.891 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-26T04:05:56.891 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000188767 s, 2.7 MB/s 2024-03-26T04:05:56.892 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:05:56.947 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_4 2024-03-26T04:05:57.002 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:05:57.002 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:57.002 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T04:05:57.002 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:57.002 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:57.002 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-26 04:05:49.357324235 +0000 2024-03-26T04:05:57.002 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-26 04:05:15.411977761 +0000 2024-03-26T04:05:57.002 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-26 04:05:15.411977761 +0000 2024-03-26T04:05:57.002 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-26 04:05:15.411977761 +0000 2024-03-26T04:05:57.003 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:05:57.067 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-26T04:05:57.067 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-26T04:05:57.067 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.00022361 s, 2.3 MB/s 2024-03-26T04:05:57.068 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:05:57.125 DEBUG:teuthology.orchestra.run.smithi149:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T04:05:57.274 INFO:teuthology.orchestra.run.smithi149.stdout:loop 2024-03-26T04:05:57.275 INFO:tasks.nvme_loop:Connecting nvme_loop smithi149:/dev/vg_nvme/lv_1... 2024-03-26T04:05:57.275 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T04:05:57.314 INFO:teuthology.orchestra.run.smithi149.stdout:1 2024-03-26T04:05:57.353 INFO:teuthology.orchestra.run.smithi149.stdout:/dev/vg_nvme/lv_11 2024-03-26T04:05:57.382 INFO:tasks.nvme_loop:Connecting nvme_loop smithi149:/dev/vg_nvme/lv_2... 2024-03-26T04:05:57.382 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T04:05:57.462 INFO:teuthology.orchestra.run.smithi149.stdout:1 2024-03-26T04:05:57.500 INFO:teuthology.orchestra.run.smithi149.stdout:/dev/vg_nvme/lv_21 2024-03-26T04:05:57.536 INFO:tasks.nvme_loop:Connecting nvme_loop smithi149:/dev/vg_nvme/lv_3... 2024-03-26T04:05:57.536 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T04:05:57.614 INFO:teuthology.orchestra.run.smithi149.stdout:1 2024-03-26T04:05:57.651 INFO:teuthology.orchestra.run.smithi149.stdout:/dev/vg_nvme/lv_31 2024-03-26T04:05:57.679 INFO:tasks.nvme_loop:Connecting nvme_loop smithi149:/dev/vg_nvme/lv_4... 2024-03-26T04:05:57.679 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T04:05:57.759 INFO:teuthology.orchestra.run.smithi149.stdout:1 2024-03-26T04:05:57.797 INFO:teuthology.orchestra.run.smithi149.stdout:/dev/vg_nvme/lv_41 2024-03-26T04:05:57.825 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:05:57.825 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:05:57.886 DEBUG:teuthology.orchestra.run.smithi149:> sudo nvme list 2024-03-26T04:05:57.951 INFO:teuthology.orchestra.run.smithi149.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T04:05:57.951 INFO:teuthology.orchestra.run.smithi149.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T04:05:57.951 INFO:teuthology.orchestra.run.smithi149.stdout:/dev/nvme0n1 CVFT623300CK400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-26T04:05:57.951 INFO:teuthology.orchestra.run.smithi149.stdout:/dev/nvme1n1 9c2891eac819482b3938 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:57.951 INFO:teuthology.orchestra.run.smithi149.stdout:/dev/nvme2n1 5cdf5ef71f9776f85b8a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:57.951 INFO:teuthology.orchestra.run.smithi149.stdout:/dev/nvme3n1 9dd9cdac63af18ec9d85 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:57.951 INFO:teuthology.orchestra.run.smithi149.stdout:/dev/nvme4n1 17aeb1fa853bb1826a22 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:57.952 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:05:57.952 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:05:57.953 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/scratch_devs 2024-03-26T04:05:58.019 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:05:58.019 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:05:58.038 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:05:58.038 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-03-26T04:05:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:05:58.095 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:58.095 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T04:05:58.095 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:58.095 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:58.095 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-26 04:05:49.389355301 +0000 2024-03-26T04:05:58.095 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-26 04:05:14.741848054 +0000 2024-03-26T04:05:58.096 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-26 04:05:14.741848054 +0000 2024-03-26T04:05:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-26 04:05:14.741848054 +0000 2024-03-26T04:05:58.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:05:58.165 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-26T04:05:58.165 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-26T04:05:58.165 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000274833 s, 1.9 MB/s 2024-03-26T04:05:58.167 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:05:58.226 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-03-26T04:05:58.283 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:05:58.283 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:58.283 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T04:05:58.283 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:58.283 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:58.283 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-26 04:05:49.389355301 +0000 2024-03-26T04:05:58.284 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-26 04:05:15.090843091 +0000 2024-03-26T04:05:58.284 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-26 04:05:15.090843091 +0000 2024-03-26T04:05:58.284 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-26 04:05:15.090843091 +0000 2024-03-26T04:05:58.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:05:58.351 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-26T04:05:58.351 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-26T04:05:58.351 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000197939 s, 2.6 MB/s 2024-03-26T04:05:58.352 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:05:58.410 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-03-26T04:05:58.466 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:05:58.467 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-26T04:05:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-26 04:05:49.389355301 +0000 2024-03-26T04:05:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-26 04:05:15.433838213 +0000 2024-03-26T04:05:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-26 04:05:15.433838213 +0000 2024-03-26T04:05:58.467 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-26 04:05:15.433838213 +0000 2024-03-26T04:05:58.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:05:58.534 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-26T04:05:58.534 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-26T04:05:58.535 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000235206 s, 2.2 MB/s 2024-03-26T04:05:58.536 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:05:58.592 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-03-26T04:05:58.649 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:05:58.649 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:05:58.649 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-26T04:05:58.649 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:05:58.649 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:05:58.649 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-26 04:05:49.389355301 +0000 2024-03-26T04:05:58.649 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-26 04:05:15.788833164 +0000 2024-03-26T04:05:58.649 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-26 04:05:15.788833164 +0000 2024-03-26T04:05:58.649 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-26 04:05:15.788833164 +0000 2024-03-26T04:05:58.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:05:58.716 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-26T04:05:58.716 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-26T04:05:58.716 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000183891 s, 2.8 MB/s 2024-03-26T04:05:58.718 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:05:58.775 DEBUG:teuthology.orchestra.run.smithi204:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T04:05:58.923 INFO:teuthology.orchestra.run.smithi204.stdout:loop 2024-03-26T04:05:58.925 INFO:tasks.nvme_loop:Connecting nvme_loop smithi204:/dev/vg_nvme/lv_1... 2024-03-26T04:05:58.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T04:05:58.962 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-26T04:05:58.999 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/vg_nvme/lv_11 2024-03-26T04:05:59.029 INFO:tasks.nvme_loop:Connecting nvme_loop smithi204:/dev/vg_nvme/lv_2... 2024-03-26T04:05:59.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T04:05:59.109 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-26T04:05:59.148 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/vg_nvme/lv_21 2024-03-26T04:05:59.177 INFO:tasks.nvme_loop:Connecting nvme_loop smithi204:/dev/vg_nvme/lv_3... 2024-03-26T04:05:59.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T04:05:59.256 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-26T04:05:59.295 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/vg_nvme/lv_31 2024-03-26T04:05:59.324 INFO:tasks.nvme_loop:Connecting nvme_loop smithi204:/dev/vg_nvme/lv_4... 2024-03-26T04:05:59.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T04:05:59.403 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-26T04:05:59.442 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/vg_nvme/lv_41 2024-03-26T04:05:59.472 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:05:59.473 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:05:59.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo nvme list 2024-03-26T04:05:59.596 INFO:teuthology.orchestra.run.smithi204.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T04:05:59.596 INFO:teuthology.orchestra.run.smithi204.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T04:05:59.596 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/nvme0n1 CVFT623300GD400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-26T04:05:59.597 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/nvme1n1 34c6d14bf58602f019ac Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:59.597 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/nvme2n1 129627ed0070849689e9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:59.597 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/nvme3n1 e66e068177209b774f9d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:59.597 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/nvme4n1 ebaae01b9eebefd29b2c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T04:05:59.598 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:05:59.598 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:05:59.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/scratch_devs 2024-03-26T04:05:59.666 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T04:05:59.760 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T04:05:59.761 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:05:59.761 INFO:tasks.cephadm:Cluster fsid is 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:05:59.761 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T04:05:59.761 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.136', 'mon.b': '172.21.15.149', 'mon.c': '172.21.15.204'} 2024-03-26T04:05:59.762 INFO:tasks.cephadm:First mon is mon.a on smithi136 2024-03-26T04:05:59.762 INFO:tasks.cephadm:First mgr is y 2024-03-26T04:05:59.762 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T04:05:59.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2024-03-26T04:05:59.795 DEBUG:teuthology.orchestra.run.smithi149:> sudo hostname $(hostname -s) 2024-03-26T04:05:59.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname $(hostname -s) 2024-03-26T04:05:59.854 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-26T04:05:59.855 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:06:00.071 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54690/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1775-g5e70124b', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1775.g5e70124b'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-25 18:23:16.641218', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-26T04:06:00.288 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 5e70124b46c2a671efcaae4690cd9ea72670710a from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:06:00.290 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm 2024-03-26T04:06:00.290 DEBUG:teuthology.orchestra.run.smithi136:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:06:00.759 INFO:teuthology.orchestra.run.smithi136.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 04:06 /home/ubuntu/cephtest/cephadm 2024-03-26T04:06:00.759 DEBUG:teuthology.orchestra.run.smithi149:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:06:01.409 INFO:teuthology.orchestra.run.smithi149.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 04:06 /home/ubuntu/cephtest/cephadm 2024-03-26T04:06:01.409 DEBUG:teuthology.orchestra.run.smithi204:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:06:01.925 INFO:teuthology.orchestra.run.smithi204.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 04:06 /home/ubuntu/cephtest/cephadm 2024-03-26T04:06:01.925 DEBUG:teuthology.orchestra.run.smithi136:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:06:01.944 DEBUG:teuthology.orchestra.run.smithi149:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:06:01.963 DEBUG:teuthology.orchestra.run.smithi204:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:06:01.994 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on all hosts... 2024-03-26T04:06:01.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:06:01.997 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:06:02.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:06:02.792 INFO:teuthology.orchestra.run.smithi204.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:06:02.827 INFO:teuthology.orchestra.run.smithi149.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:06:02.835 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:06:40.769 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-03-26T04:06:40.770 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:06:40.770 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:06:40.770 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2024-03-26T04:06:40.770 INFO:teuthology.orchestra.run.smithi136.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:06:40.770 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-03-26T04:06:40.770 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-03-26T04:06:44.093 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-26T04:06:44.093 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:06:44.093 INFO:teuthology.orchestra.run.smithi204.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:06:44.093 INFO:teuthology.orchestra.run.smithi204.stdout: "repo_digests": [ 2024-03-26T04:06:44.093 INFO:teuthology.orchestra.run.smithi204.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:06:44.093 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-26T04:06:44.093 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-26T04:06:52.815 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-03-26T04:06:52.815 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:06:52.815 INFO:teuthology.orchestra.run.smithi149.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:06:52.815 INFO:teuthology.orchestra.run.smithi149.stdout: "repo_digests": [ 2024-03-26T04:06:52.815 INFO:teuthology.orchestra.run.smithi149.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:06:52.815 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-03-26T04:06:52.815 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-03-26T04:06:52.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2024-03-26T04:06:52.874 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph 2024-03-26T04:06:52.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph 2024-03-26T04:06:52.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2024-03-26T04:06:52.971 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 777 /etc/ceph 2024-03-26T04:06:52.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /etc/ceph 2024-03-26T04:06:53.029 INFO:tasks.cephadm:Writing seed config... 2024-03-26T04:06:53.030 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T04:06:53.030 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T04:06:53.030 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-26T04:06:53.030 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-26T04:06:53.030 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T04:06:53.030 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T04:06:53.030 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T04:06:53.030 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T04:06:53.030 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T04:06:53.031 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T04:06:53.031 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:06:53.031 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T04:06:53.048 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 276a1e6c-eb26-11ee-95d0-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-26T04:06:53.048 DEBUG:teuthology.orchestra.run.smithi136:mon.a> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T04:06:53.091 DEBUG:teuthology.orchestra.run.smithi136:mgr.y> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T04:06:53.134 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T04:06:53.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a -v bootstrap --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.136 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:06:53.328 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:06:53.328 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a', '-v', 'bootstrap', '--fsid', '276a1e6c-eb26-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.136', '--skip-admin-label'] 2024-03-26T04:06:53.355 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:06:53.355 INFO:teuthology.orchestra.run.smithi136.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T04:06:53.355 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying podman|docker is present... 2024-03-26T04:06:53.379 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:06:53.379 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying lvm2 is present... 2024-03-26T04:06:53.379 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying time synchronization is in place... 2024-03-26T04:06:53.386 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T04:06:53.386 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:06:53.392 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T04:06:53.392 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T04:06:53.398 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout enabled 2024-03-26T04:06:53.404 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout active 2024-03-26T04:06:53.404 INFO:teuthology.orchestra.run.smithi136.stdout:Unit chronyd.service is enabled and running 2024-03-26T04:06:53.404 INFO:teuthology.orchestra.run.smithi136.stdout:Repeating the final host check... 2024-03-26T04:06:53.427 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:06:53.427 INFO:teuthology.orchestra.run.smithi136.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T04:06:53.427 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl is present 2024-03-26T04:06:53.427 INFO:teuthology.orchestra.run.smithi136.stdout:lvcreate is present 2024-03-26T04:06:53.433 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T04:06:53.433 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:06:53.439 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T04:06:53.439 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T04:06:53.445 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout enabled 2024-03-26T04:06:53.451 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout active 2024-03-26T04:06:53.451 INFO:teuthology.orchestra.run.smithi136.stdout:Unit chronyd.service is enabled and running 2024-03-26T04:06:53.451 INFO:teuthology.orchestra.run.smithi136.stdout:Host looks OK 2024-03-26T04:06:53.451 INFO:teuthology.orchestra.run.smithi136.stdout:Cluster fsid: 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:06:53.451 INFO:teuthology.orchestra.run.smithi136.stdout:Acquiring lock 140536225725024 on /run/cephadm/276a1e6c-eb26-11ee-95d0-87774f69a715.lock 2024-03-26T04:06:53.452 INFO:teuthology.orchestra.run.smithi136.stdout:Lock 140536225725024 acquired on /run/cephadm/276a1e6c-eb26-11ee-95d0-87774f69a715.lock 2024-03-26T04:06:53.452 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying IP 172.21.15.136 port 3300 ... 2024-03-26T04:06:53.452 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying IP 172.21.15.136 port 6789 ... 2024-03-26T04:06:53.452 INFO:teuthology.orchestra.run.smithi136.stdout:Base mon IP(s) is [172.21.15.136:3300, 172.21.15.136:6789], mon addrv is [v2:172.21.15.136:3300,v1:172.21.15.136:6789] 2024-03-26T04:06:53.455 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.136 metric 100 2024-03-26T04:06:53.455 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.136 metric 100 2024-03-26T04:06:53.458 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T04:06:53.458 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-26T04:06:53.458 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1771sec hoplimit 64 pref medium 2024-03-26T04:06:53.460 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T04:06:53.460 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T04:06:53.460 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T04:06:53.460 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-26T04:06:53.460 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6e43/64 scope link 2024-03-26T04:06:53.461 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T04:06:53.461 INFO:teuthology.orchestra.run.smithi136.stdout:Mon IP `172.21.15.136` is in CIDR network `172.21.0.0/20` 2024-03-26T04:06:53.461 INFO:teuthology.orchestra.run.smithi136.stdout:Mon IP `172.21.15.136` is in CIDR network `172.21.0.0/20` 2024-03-26T04:06:53.461 INFO:teuthology.orchestra.run.smithi136.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T04:06:53.461 INFO:teuthology.orchestra.run.smithi136.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T04:06:53.462 INFO:teuthology.orchestra.run.smithi136.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:06:53.925 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:06:53.925 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:06:53.925 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T04:06:53.925 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Copying blob sha256:45ddbf13772d0d6535ecfdcaa58ecfc496e5849512dfbf602c5218332a216bcc 2024-03-26T04:06:53.925 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T04:06:53.925 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Copying config sha256:5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:06:53.925 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T04:06:54.880 INFO:teuthology.orchestra.run.smithi136.stdout:ceph: stdout ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T04:06:54.880 INFO:teuthology.orchestra.run.smithi136.stdout:Ceph version: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T04:06:54.880 INFO:teuthology.orchestra.run.smithi136.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T04:06:55.993 INFO:teuthology.orchestra.run.smithi136.stdout:stat: stdout 167 167 2024-03-26T04:06:55.993 INFO:teuthology.orchestra.run.smithi136.stdout:Creating initial keys... 2024-03-26T04:06:56.857 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph-authtool: stdout AQDgSQJmd96vExAA/9ncuo9MAyope7UWCvZAzQ== 2024-03-26T04:07:04.737 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph-authtool: stdout AQDiSQJmsh87DRAA3tM0u5jgeFLmpTPEjeWy1w== 2024-03-26T04:07:06.941 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph-authtool: stdout AQDqSQJmHNWhFRAAlPU7mdv+18hpSqgu630nbw== 2024-03-26T04:07:06.941 INFO:teuthology.orchestra.run.smithi136.stdout:Creating initial monmap... 2024-03-26T04:07:07.854 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:07:07.855 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T04:07:07.855 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:07:07.855 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:07:07.855 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool for a [v2:172.21.15.136:3300,v1:172.21.15.136:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:07:07.855 INFO:teuthology.orchestra.run.smithi136.stdout:setting min_mon_release = quincy 2024-03-26T04:07:07.855 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: set fsid to 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:07:07.855 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:07:07.855 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:07:07.855 INFO:teuthology.orchestra.run.smithi136.stdout:Creating mon... 2024-03-26T04:07:08.810 INFO:teuthology.orchestra.run.smithi136.stdout:create mon.a on 2024-03-26T04:07:09.177 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T04:07:09.371 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-276a1e6c-eb26-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-276a1e6c-eb26-11ee-95d0-87774f69a715.target. 2024-03-26T04:07:09.372 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-276a1e6c-eb26-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-276a1e6c-eb26-11ee-95d0-87774f69a715.target. 2024-03-26T04:07:09.600 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.a 2024-03-26T04:07:09.600 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to reset failed state of unit ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T04:07:09.766 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-276a1e6c-eb26-11ee-95d0-87774f69a715.target.wants/ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@.service. 2024-03-26T04:07:10.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:09 smithi136 podman[25320]: 2024-03-26 04:07:09.894570459 +0000 UTC m=+0.017596596 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:07:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:10 smithi136 podman[25320]: 2024-03-26 04:07:10.360582609 +0000 UTC m=+0.483608788 container create 25c79f6481fa1ee93a7c5c98505ed7c7f16c9c44667ce9dfa7fc0efbd89d173d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:07:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:10 smithi136 podman[25320]: 2024-03-26 04:07:10.55447641 +0000 UTC m=+0.677502547 container init 25c79f6481fa1ee93a7c5c98505ed7c7f16c9c44667ce9dfa7fc0efbd89d173d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:07:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:10 smithi136 podman[25320]: 2024-03-26 04:07:10.560439536 +0000 UTC m=+0.683465674 container start 25c79f6481fa1ee93a7c5c98505ed7c7f16c9c44667ce9dfa7fc0efbd89d173d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:07:10.677 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:07:10.677 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-03-26T04:07:10.684 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:07:10.684 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T04:07:10.684 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-03-26T04:07:10.684 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T04:07:10.684 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mon to start... 2024-03-26T04:07:10.684 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mon... 2024-03-26T04:07:10.917 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:10 smithi136 ceph-mon[25358]: mkfs 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:07:10.917 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:10 smithi136 ceph-mon[25358]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:07:10.917 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:10 smithi136 bash[25320]: 25c79f6481fa1ee93a7c5c98505ed7c7f16c9c44667ce9dfa7fc0efbd89d173d 2024-03-26T04:07:10.917 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:10 smithi136 systemd[1]: Started Ceph mon.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:07:11.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:11 smithi136 ceph-mon[25358]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:07:11.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:11 smithi136 ceph-mon[25358]: monmap e1: 1 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:07:11.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:11 smithi136 ceph-mon[25358]: fsmap 2024-03-26T04:07:11.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:11 smithi136 ceph-mon[25358]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T04:07:11.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:11 smithi136 ceph-mon[25358]: mgrmap e1: no daemons active 2024-03-26T04:07:11.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:11 smithi136 ceph-mon[25358]: from='client.? 172.21.15.136:0/47390175' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T04:07:13.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T04:07:13.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout id: 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:07:13.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T04:07:13.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:13.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout services: 2024-03-26T04:07:13.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.624086s) 2024-03-26T04:07:13.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T04:07:13.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T04:07:13.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:13.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout data: 2024-03-26T04:07:13.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T04:07:13.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T04:07:13.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T04:07:13.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T04:07:13.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:13.304 INFO:teuthology.orchestra.run.smithi136.stdout:mon is available 2024-03-26T04:07:13.304 INFO:teuthology.orchestra.run.smithi136.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T04:07:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:13 smithi136 ceph-mon[25358]: from='client.? 172.21.15.136:0/1132705377' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:07:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:13 smithi136 ceph-mon[25358]: from='client.? 172.21.15.136:0/1132705377' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T04:07:14.401 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:14.401 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T04:07:14.401 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout fsid = 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:07:14.401 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.136:3300,v1:172.21.15.136:6789] 2024-03-26T04:07:14.401 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T04:07:14.401 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T04:07:14.401 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T04:07:14.401 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T04:07:14.401 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:14.402 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T04:07:14.402 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-03-26T04:07:14.402 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T04:07:14.402 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:14.402 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T04:07:14.402 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T04:07:14.402 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T04:07:14.402 INFO:teuthology.orchestra.run.smithi136.stdout:Generating new minimal ceph.conf... 2024-03-26T04:07:15.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:14 smithi136 ceph-mon[25358]: from='client.? 172.21.15.136:0/973419740' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:07:15.471 INFO:teuthology.orchestra.run.smithi136.stdout:Restarting the monitor... 2024-03-26T04:07:15.742 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:15 smithi136 systemd[1]: Stopping Ceph mon.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:07:16.180 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:15 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a[25335]: 2024-03-26T04:07:15.741+0000 7f7ed47f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:07:16.180 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:15 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a[25335]: 2024-03-26T04:07:15.741+0000 7f7ed47f2700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T04:07:16.180 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:15 smithi136 podman[25693]: 2024-03-26 04:07:15.895024024 +0000 UTC m=+0.259184024 container died 25c79f6481fa1ee93a7c5c98505ed7c7f16c9c44667ce9dfa7fc0efbd89d173d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:07:16.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:16 smithi136 podman[25693]: 2024-03-26 04:07:16.271468023 +0000 UTC m=+0.635628021 container cleanup 25c79f6481fa1ee93a7c5c98505ed7c7f16c9c44667ce9dfa7fc0efbd89d173d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:07:16.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:16 smithi136 bash[25693]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:07:16.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:16 smithi136 podman[25712]: 2024-03-26 04:07:16.547508819 +0000 UTC m=+0.650739398 container remove 25c79f6481fa1ee93a7c5c98505ed7c7f16c9c44667ce9dfa7fc0efbd89d173d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux ) 2024-03-26T04:07:16.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:16 smithi136 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T04:07:16.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:16 smithi136 systemd[1]: Stopped Ceph mon.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:07:16.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:16 smithi136 systemd[1]: Starting Ceph mon.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:07:17.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:16 smithi136 podman[25810]: 2024-03-26 04:07:16.81490701 +0000 UTC m=+0.017287116 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:07:17.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:16 smithi136 podman[25810]: 2024-03-26 04:07:16.923254073 +0000 UTC m=+0.125634151 container create b6196b95c1e4cf89b7adc52f127efcc5f708a5349fb7801cbb6c42b8f4f657df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:07:17.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 podman[25810]: 2024-03-26 04:07:17.109181914 +0000 UTC m=+0.311561999 container init b6196b95c1e4cf89b7adc52f127efcc5f708a5349fb7801cbb6c42b8f4f657df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-03-26T04:07:17.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 podman[25810]: 2024-03-26 04:07:17.114679119 +0000 UTC m=+0.317059197 container start b6196b95c1e4cf89b7adc52f127efcc5f708a5349fb7801cbb6c42b8f4f657df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T04:07:17.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:07:17.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T04:07:17.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: pidfile_write: ignore empty --pid-file 2024-03-26T04:07:17.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: load: jerasure load: lrc 2024-03-26T04:07:17.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:07:17.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Git sha 0 2024-03-26T04:07:17.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:07:17.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: DB SUMMARY 2024-03-26T04:07:17.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: DB Session ID: YDGGB1TAE1J8KKYY8WEG 2024-03-26T04:07:17.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:07:17.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:07:17.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T04:07:17.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T04:07:17.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84645 ; 2024-03-26T04:07:17.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:07:17.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:07:17.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:07:17.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:07:17.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:07:17.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.env: 0x55d17fdafae0 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.info_log: 0x55d1815f9920 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.statistics: (nil) 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.use_fsync: 0 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:07:17.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.db_log_dir: 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.wal_dir: 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:07:17.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.write_buffer_manager: 0x55d1815cc1e0 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.unordered_write: 0 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:07:17.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.row_cache: None 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.wal_filter: None 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.wal_compression: 0 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:07:17.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:07:17.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_open_files: -1 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Compression algorithms supported: 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:07:17.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: kZSTD supported: 0 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.merge_operator: 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_filter: None 2024-03-26T04:07:17.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d1815f9a20) 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_type: 0 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout: data_block_index_type: 0 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_shortening: 1 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout: checksum: 4 2024-03-26T04:07:17.194 INFO:journalctl@ceph.mon.a.smithi136.stdout: no_block_cache: 0 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache: 0x55d1815fd090 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_options: 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: capacity : 536870912 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: num_shard_bits : 4 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: strict_capacity_limit : 0 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_compressed: (nil) 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: persistent_cache: (nil) 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_size: 4096 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_size_deviation: 10 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_restart_interval: 16 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_block_restart_interval: 1 2024-03-26T04:07:17.195 INFO:journalctl@ceph.mon.a.smithi136.stdout: metadata_block_size: 4096 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: partition_filters: 0 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: use_delta_encoding: 1 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: filter_policy: bloomfilter 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: whole_key_filtering: 1 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: verify_compression: 0 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: format_version: 5 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: enable_index_compression: 1 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_align: 0 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: max_auto_readahead_size: 262144 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: prepopulate_block_cache: 0 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:07:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compression: NoCompression 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.num_levels: 7 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:07:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:07:17.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:07:17.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:07:17.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:07:17.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:07:17.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:07:17.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:07:17.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:07:17.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:07:17.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:07:17.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:07:17.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:07:17.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:07:17.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:07:17.202 INFO:teuthology.orchestra.run.smithi136.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T04:07:17.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.ttl: 2592000 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:07:17.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 59ce1048-0cd7-4c08-864a-a7c8b13474a3 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426037143625, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T04:07:17.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426037144465, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81270, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79457, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73766, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711426037, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "59ce1048-0cd7-4c08-864a-a7c8b13474a3", "db_session_id": "YDGGB1TAE1J8KKYY8WEG", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426037144548, "job": 1, "event": "recovery_finished"} 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d1816ee000 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: DB pointer 0x55d1816d8000 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** DB Stats ** 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:07:17.205 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** Compaction Stats [default] ** 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: L0 2/0 81.20 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: Sum 2/0 81.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** Compaction Stats [default] ** 2024-03-26T04:07:17.206 INFO:journalctl@ceph.mon.a.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative compaction: 0.00 GB write, 28.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval compaction: 0.00 GB write, 28.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:07:17.207 INFO:journalctl@ceph.mon.a.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout: Block cache BinnedLRUCache@0x55d1815fd090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: starting mon.a rank 0 at public addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] at bind addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: mon.a@-1(???) e1 preinit fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: mon.a@-1(???).mds e1 new map 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: mon.a@-1(???).mds e1 print_map 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout: e1 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout: legacy client fscid: -1 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout: No filesystems configured 2024-03-26T04:07:17.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:07:17.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:07:17.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:07:17.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:07:17.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T04:07:17.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 bash[25810]: b6196b95c1e4cf89b7adc52f127efcc5f708a5349fb7801cbb6c42b8f4f657df 2024-03-26T04:07:17.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 systemd[1]: Started Ceph mon.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:07:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:07:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: monmap e1: 1 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:07:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: fsmap 2024-03-26T04:07:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T04:07:17.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:17 smithi136 ceph-mon[25848]: mgrmap e1: no daemons active 2024-03-26T04:07:18.309 INFO:teuthology.orchestra.run.smithi136.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T04:07:18.331 INFO:teuthology.orchestra.run.smithi136.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:07:18.331 INFO:teuthology.orchestra.run.smithi136.stdout:Creating mgr... 2024-03-26T04:07:18.332 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T04:07:18.332 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T04:07:18.576 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.y 2024-03-26T04:07:18.576 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to reset failed state of unit ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T04:07:18.731 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-276a1e6c-eb26-11ee-95d0-87774f69a715.target.wants/ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@.service. 2024-03-26T04:07:18.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:18 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/717855546' entity='client.admin' 2024-03-26T04:07:18.949 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:18 smithi136 systemd[1]: Starting Ceph mgr.y for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:07:19.303 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:18 smithi136 podman[26087]: 2024-03-26 04:07:18.968633392 +0000 UTC m=+0.113193044 container create 06ee71789a25d6a906d05cee3000f11e71a06f1a2788c6bffb1a7bae4a4771e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:07:19.303 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:18 smithi136 podman[26087]: 2024-03-26 04:07:18.87313896 +0000 UTC m=+0.017698613 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:07:19.303 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:19 smithi136 podman[26087]: 2024-03-26 04:07:19.178356284 +0000 UTC m=+0.322915936 container init 06ee71789a25d6a906d05cee3000f11e71a06f1a2788c6bffb1a7bae4a4771e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T04:07:19.303 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:19 smithi136 podman[26087]: 2024-03-26 04:07:19.18411045 +0000 UTC m=+0.328670104 container start 06ee71789a25d6a906d05cee3000f11e71a06f1a2788c6bffb1a7bae4a4771e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-26T04:07:19.303 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:19 smithi136 bash[26087]: 06ee71789a25d6a906d05cee3000f11e71a06f1a2788c6bffb1a7bae4a4771e8 2024-03-26T04:07:19.316 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:07:19.316 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-03-26T04:07:19.324 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:07:19.324 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T04:07:19.324 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-03-26T04:07:19.324 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T04:07:19.332 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:07:19.332 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-03-26T04:07:19.340 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:07:19.340 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T04:07:19.340 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-03-26T04:07:19.340 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T04:07:19.340 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr to start... 2024-03-26T04:07:19.340 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr... 2024-03-26T04:07:19.580 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:19 smithi136 systemd[1]: Started Ceph mgr.y for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:07:19.830 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:19 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:19.736+0000 7f4bcd609200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:07:19.830 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:19 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:19.830+0000 7f4bcd609200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:07:20.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:20 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1462375428' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsid": "276a1e6c-eb26-11ee-95d0-87774f69a715", 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:07:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:07:20.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:07:20.745 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:07:20.745 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:07:20.745 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:07:20.745 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:20.745 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:07:20.745 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:07:20.745 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:07:20.745 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:07:20.746 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:07:10.618955+0000", 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:07:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:mgr not available, waiting (1/15)... 2024-03-26T04:07:20.822 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:20 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:20.491+0000 7f4bcd609200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:07:20.822 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:20 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:20.582+0000 7f4bcd609200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:07:20.822 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:20 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:20.725+0000 7f4bcd609200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:07:21.173 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:20 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:20.821+0000 7f4bcd609200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:07:21.174 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:20 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:20.903+0000 7f4bcd609200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:07:21.174 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:21 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:21.173+0000 7f4bcd609200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:07:21.448 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:21 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:21.281+0000 7f4bcd609200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:07:21.449 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:21 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:21.366+0000 7f4bcd609200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:07:21.895 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:21 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:21.459+0000 7f4bcd609200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:07:21.895 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:21 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:21.634+0000 7f4bcd609200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:07:22.736 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:22 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:22.607+0000 7f4bcd609200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:07:22.736 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:22 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:22.735+0000 7f4bcd609200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:07:23.300 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:23 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:23.079+0000 7f4bcd609200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:07:23.300 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:23 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:23.171+0000 7f4bcd609200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:07:23.563 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:23 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/100429519' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:07:23.563 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:23 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:23.377+0000 7f4bcd609200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:07:23.841 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:23 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:23.562+0000 7f4bcd609200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsid": "276a1e6c-eb26-11ee-95d0-87774f69a715", 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:07:23.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:07:23.891 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:07:23.891 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:23.891 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:07:23.891 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:07:23.891 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:07:23.894 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:07:10.618955+0000", 2024-03-26T04:07:23.895 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:07:23.896 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:23.896 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:07:23.896 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:07:23.896 INFO:teuthology.orchestra.run.smithi136.stdout:mgr not available, waiting (2/15)... 2024-03-26T04:07:24.108 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:23 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:23.847+0000 7f4bcd609200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:07:24.108 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:24 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:24.108+0000 7f4bcd609200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:07:24.448 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:24 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:24.342+0000 7f4bcd609200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:07:25.448 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:25 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:25.133+0000 7f4bcd609200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:07:25.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:25 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:25.694+0000 7f4bcd609200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:07:25.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:25 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:25.790+0000 7f4bcd609200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:07:26.301 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:26.007+0000 7f4bcd609200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: Activating manager daemon y 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: mgrmap e2: y(active, starting, since 0.0030162s) 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: from='mgr.14100 172.21.15.136:0/2080441248' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: from='mgr.14100 172.21.15.136:0/2080441248' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: from='mgr.14100 172.21.15.136:0/2080441248' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: from='mgr.14100 172.21.15.136:0/2080441248' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: from='mgr.14100 172.21.15.136:0/2080441248' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: Manager daemon y is now available 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: from='mgr.14100 172.21.15.136:0/2080441248' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: from='mgr.14100 172.21.15.136:0/2080441248' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: from='mgr.14100 172.21.15.136:0/2080441248' entity='mgr.y' 2024-03-26T04:07:26.302 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: from='mgr.14100 172.21.15.136:0/2080441248' entity='mgr.y' 2024-03-26T04:07:26.303 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:26 smithi136 ceph-mon[25848]: from='mgr.14100 172.21.15.136:0/2080441248' entity='mgr.y' 2024-03-26T04:07:27.027 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:27.027 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:07:27.027 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsid": "276a1e6c-eb26-11ee-95d0-87774f69a715", 2024-03-26T04:07:27.027 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:07:27.027 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:07:27.027 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:07:27.027 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:07:27.027 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:27.027 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:07:27.027 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:27.028 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:07:27.029 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:07:27.029 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:07:27.029 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:07:27.029 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:07:27.029 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:07:27.032 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:27.032 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:07:27.032 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:07:27.032 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:07:27.032 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:07:27.032 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:07:27.032 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:07:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:07:10.618955+0000", 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:07:27.034 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:07:27.035 INFO:teuthology.orchestra.run.smithi136.stdout:mgr not available, waiting (3/15)... 2024-03-26T04:07:27.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:27 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1440638282' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:07:27.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:27 smithi136 ceph-mon[25848]: mgrmap e3: y(active, since 1.00783s) 2024-03-26T04:07:29.441 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:29 smithi136 ceph-mon[25848]: mgrmap e4: y(active, since 2s) 2024-03-26T04:07:30.419 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:30 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/3403996837' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:07:30.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:30.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:07:30.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsid": "276a1e6c-eb26-11ee-95d0-87774f69a715", 2024-03-26T04:07:30.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:07:30.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:07:30.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:07:30.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:30.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:07:30.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:30.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:07:30.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:07:30.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:07:30.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:07:30.429 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:07:30.429 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:07:30.429 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:30.429 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:30.430 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:07:10.618955+0000", 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:07:30.431 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:07:30.432 INFO:teuthology.orchestra.run.smithi136.stdout:mgr is available 2024-03-26T04:07:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:31 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1576582714' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:07:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:31 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1576582714' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T04:07:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T04:07:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout fsid = 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:07:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T04:07:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T04:07:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T04:07:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T04:07:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T04:07:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T04:07:31.614 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T04:07:31.614 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T04:07:31.614 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T04:07:31.614 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T04:07:31.614 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling cephadm module... 2024-03-26T04:07:32.657 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:32 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/3379070980' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T04:07:32.657 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:32 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ignoring --setuser ceph since I am not root 2024-03-26T04:07:32.657 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:32 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ignoring --setgroup ceph since I am not root 2024-03-26T04:07:33.168 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:32 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:32.910+0000 7fccf6cff200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:07:33.168 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:33 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:33.013+0000 7fccf6cff200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:07:33.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:33 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/3379070980' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T04:07:33.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:33 smithi136 ceph-mon[25848]: mgrmap e5: y(active, since 6s) 2024-03-26T04:07:33.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:33 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:33.668+0000 7fccf6cff200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:07:33.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:33 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:33.765+0000 7fccf6cff200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:07:33.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:33 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:33.908+0000 7fccf6cff200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:07:34.207 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:07:34.208 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T04:07:34.208 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:07:34.208 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T04:07:34.208 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T04:07:34.208 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:07:34.208 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for the mgr to restart... 2024-03-26T04:07:34.208 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr epoch 5... 2024-03-26T04:07:34.329 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:34 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:34.005+0000 7fccf6cff200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:07:34.329 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:34 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:34.088+0000 7fccf6cff200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:07:34.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:34 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1239742407' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:07:34.580 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:34 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:34.359+0000 7fccf6cff200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:07:34.580 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:34 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:34.473+0000 7fccf6cff200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:07:34.580 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:34 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:34.560+0000 7fccf6cff200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:07:34.838 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:34 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:34.659+0000 7fccf6cff200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:07:35.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:34 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:34.838+0000 7fccf6cff200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:07:36.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:35 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:35.789+0000 7fccf6cff200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:07:36.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:35 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:35.918+0000 7fccf6cff200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:07:36.535 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:36 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:36.251+0000 7fccf6cff200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:07:36.535 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:36 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:36.340+0000 7fccf6cff200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:07:36.536 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:36 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:36.535+0000 7fccf6cff200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:07:36.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:36 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:36.719+0000 7fccf6cff200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:07:37.262 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:37 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:37.001+0000 7fccf6cff200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:07:37.262 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:37 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:37.261+0000 7fccf6cff200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:07:37.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:37 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:37.495+0000 7fccf6cff200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:07:38.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:38 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:38.287+0000 7fccf6cff200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:07:39.164 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:38 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:38.850+0000 7fccf6cff200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:07:39.164 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:38 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:38.947+0000 7fccf6cff200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:07:39.164 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:07:39.162+0000 7fccf6cff200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:07:39.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: Active manager daemon y restarted 2024-03-26T04:07:39.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: Activating manager daemon y 2024-03-26T04:07:39.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T04:07:39.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: mgrmap e6: y(active, starting, since 0.00576348s) 2024-03-26T04:07:39.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:07:39.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:07:39.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:07:39.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:07:39.416 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:07:39.416 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: Manager daemon y is now available 2024-03-26T04:07:39.416 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:39 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:40.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: Found migration_current of "None". Setting to last migration. 2024-03-26T04:07:40.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:40.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:07:40.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:07:40.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:07:40.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:07:40.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:07:40.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:40.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:40.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: mgrmap e7: y(active, since 1.00555s) 2024-03-26T04:07:40.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:40 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:07:40.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:07:40.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T04:07:40.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T04:07:40.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:07:40.715 INFO:teuthology.orchestra.run.smithi136.stdout:mgr epoch 5 is available 2024-03-26T04:07:40.715 INFO:teuthology.orchestra.run.smithi136.stdout:Setting orchestrator backend to cephadm... 2024-03-26T04:07:41.414 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:41 smithi136 ceph-mon[25848]: [26/Mar/2024:04:07:39] ENGINE Bus STARTING 2024-03-26T04:07:41.414 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:41 smithi136 ceph-mon[25848]: [26/Mar/2024:04:07:40] ENGINE Serving on https://172.21.15.136:7150 2024-03-26T04:07:41.414 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:41 smithi136 ceph-mon[25848]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:07:41.414 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:41 smithi136 ceph-mon[25848]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:07:41.414 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:41 smithi136 ceph-mon[25848]: [26/Mar/2024:04:07:40] ENGINE Serving on http://172.21.15.136:8765 2024-03-26T04:07:41.414 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:41 smithi136 ceph-mon[25848]: [26/Mar/2024:04:07:40] ENGINE Bus STARTED 2024-03-26T04:07:42.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:42 smithi136 ceph-mon[25848]: mgrmap e8: y(active, since 2s) 2024-03-26T04:07:42.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:42 smithi136 ceph-mon[25848]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:07:42.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:42 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:42.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:42 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:07:43.150 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T04:07:43.151 INFO:teuthology.orchestra.run.smithi136.stdout:Generating ssh key... 2024-03-26T04:07:43.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:43 smithi136 ceph-mon[25848]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:07:44.337 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: Generating public/private rsa key pair. 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: Your identification has been saved in /tmp/tmp2_vna5m8/key. 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: Your public key has been saved in /tmp/tmp2_vna5m8/key.pub. 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: The key fingerprint is: 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: SHA256:F2WB9kHRcBWa0n/waiAjOKm7SolyMY8RH6QXwN5orXg ceph-276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: The key's randomart image is: 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: +---[RSA 3072]----+ 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: | ...o oB=.oo| 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: | .o . ooo.+ | 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: | .o+o ..o =. | 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: | ++o. o .o .o | 2024-03-26T04:07:44.338 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: | o+.. + S + . .o| 2024-03-26T04:07:44.339 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: |..EB . . o o . ..| 2024-03-26T04:07:44.339 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: |o.= o o | 2024-03-26T04:07:44.339 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: |.o . . | 2024-03-26T04:07:44.339 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: | ..o. | 2024-03-26T04:07:44.339 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: +----[SHA256]-----+ 2024-03-26T04:07:45.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:45 smithi136 ceph-mon[25848]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:07:45.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:45 smithi136 ceph-mon[25848]: Generating ssh key... 2024-03-26T04:07:45.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:45 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:45.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:45 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:45.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq4ec9ZZE2U5j1mWpuJvwHVWDpNRLQwmsmoJNFLM4sVfR+DoyKh3Ys0cPAzKxltgxik2w9TPJiVXwwaHdsXxRSrI6gG+qb9zSAnq4jJmitQb4ZVjXFYraIftseUYY9dxcDyKTGBdAnfev1Dug8+v/JF+QR17sc7uVkanovtEJVVeubbUd3aWaGqib59mjtCw/Fm74R6VM+Y5MTKFwZAKw0Z8e8C7tRLshUaX3YgFD5PxntQYbcanP7Mv0X1VrBjJV5bAYtC9U8sHi/gTLquFRfXFP/1vp7g7Q0cDWr9w0Bzy3hjW3y7q1LCQGKGJYpxYdpdoUAJI3UGqTQhc3GaiB+O9QVPvb3SwfEk/6wZhBWUcxGmnSkLmXX9glH1yvjIfBAwqfWS+v5x0EudQy+VW41NB7kXIq6i3MFR5e862KtdQm1Bw/HZV8bL9nC+OY4EM3bD/ZKWcNsKqeAWsSJ86+cdYilNkyZbLapb65kZMumY525MFekTrYzvOVtGI0Vedc= ceph-276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:07:45.822 INFO:teuthology.orchestra.run.smithi136.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:07:45.822 INFO:teuthology.orchestra.run.smithi136.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T04:07:45.822 INFO:teuthology.orchestra.run.smithi136.stdout:Adding host smithi136... 2024-03-26T04:07:46.180 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:46 smithi136 ceph-mon[25848]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:07:47.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:47 smithi136 ceph-mon[25848]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "addr": "172.21.15.136", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:07:47.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:47 smithi136 ceph-mon[25848]: Deploying cephadm binary to smithi136 2024-03-26T04:07:51.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Added host 'smithi136' with addr '172.21.15.136' 2024-03-26T04:07:51.279 INFO:teuthology.orchestra.run.smithi136.stdout:Deploying unmanaged mon service... 2024-03-26T04:07:51.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:51 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:51.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:51 smithi136 ceph-mon[25848]: Added host smithi136 2024-03-26T04:07:51.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:51 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:07:53.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:53 smithi136 ceph-mon[25848]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:07:53.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:53 smithi136 ceph-mon[25848]: Saving service mon spec with placement count:5 2024-03-26T04:07:53.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:53 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:53.628 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T04:07:53.628 INFO:teuthology.orchestra.run.smithi136.stdout:Deploying unmanaged mgr service... 2024-03-26T04:07:54.548 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:54 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:54.548 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:54 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:55.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:55 smithi136 ceph-mon[25848]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:07:55.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:55 smithi136 ceph-mon[25848]: Saving service mgr spec with placement count:2 2024-03-26T04:07:55.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:55 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:55.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:55 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:55.579 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T04:07:57.525 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:57 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4082309215' entity='client.admin' 2024-03-26T04:07:57.525 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:57 smithi136 ceph-mon[25848]: from='mgr.14120 172.21.15.136:0/125283448' entity='mgr.y' 2024-03-26T04:07:58.136 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling the dashboard module... 2024-03-26T04:07:58.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:58 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1860336962' entity='client.admin' 2024-03-26T04:07:59.852 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:59 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/2727507642' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T04:07:59.852 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:07:59 smithi136 ceph-mon[25848]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:07:59.852 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:59 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ignoring --setuser ceph since I am not root 2024-03-26T04:07:59.853 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:07:59 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ignoring --setgroup ceph since I am not root 2024-03-26T04:08:00.371 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:00 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:00.116+0000 7fe8b3bda200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:08:00.372 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:00 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:00.214+0000 7fe8b3bda200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:08:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:00 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/2727507642' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T04:08:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:00 smithi136 ceph-mon[25848]: mgrmap e9: y(active, since 20s) 2024-03-26T04:08:01.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:00 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:00.875+0000 7fe8b3bda200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:08:01.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:00 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:00.973+0000 7fe8b3bda200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:08:01.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:01 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:01.117+0000 7fe8b3bda200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:08:01.459 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:01 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:01.215+0000 7fe8b3bda200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:08:01.459 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:01 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:01.299+0000 7fe8b3bda200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:08:01.462 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:08:01.462 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T04:08:01.462 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:08:01.462 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T04:08:01.462 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T04:08:01.462 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:08:01.462 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for the mgr to restart... 2024-03-26T04:08:01.462 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr epoch 9... 2024-03-26T04:08:01.710 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:01 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4115648973' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:08:01.710 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:01 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:01.575+0000 7fe8b3bda200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:08:01.711 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:01 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:01.688+0000 7fe8b3bda200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:08:02.058 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:01 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:01.779+0000 7fe8b3bda200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:08:02.058 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:01 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:01.877+0000 7fe8b3bda200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:08:02.058 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:02 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:02.057+0000 7fe8b3bda200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:08:03.448 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:03.012+0000 7fe8b3bda200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:08:03.448 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:03.141+0000 7fe8b3bda200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:08:03.764 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:03.477+0000 7fe8b3bda200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:08:03.764 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:03.567+0000 7fe8b3bda200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:08:03.764 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:03.763+0000 7fe8b3bda200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:08:04.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:03.948+0000 7fe8b3bda200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:08:04.494 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:04 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:04.233+0000 7fe8b3bda200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:08:04.494 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:04 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:04.493+0000 7fe8b3bda200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:08:04.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:04 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:04.731+0000 7fe8b3bda200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:08:05.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:05 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:05.527+0000 7fe8b3bda200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:08:06.412 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:06.098+0000 7fe8b3bda200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:08:06.412 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:06.195+0000 7fe8b3bda200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:08:06.413 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:08:06.412+0000 7fe8b3bda200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:08:06.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: Active manager daemon y restarted 2024-03-26T04:08:06.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: Activating manager daemon y 2024-03-26T04:08:06.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T04:08:06.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: mgrmap e10: y(active, starting, since 0.0033273s) 2024-03-26T04:08:06.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:08:06.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:08:06.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:08:06.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:08:06.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:08:06.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: Manager daemon y is now available 2024-03-26T04:08:06.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:06.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:06 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:07.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:08:07.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:08:07.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:07 smithi136 ceph-mon[25848]: mgrmap e11: y(active, since 1.0069s) 2024-03-26T04:08:08.372 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:08:08.372 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T04:08:08.372 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T04:08:08.372 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:08:08.372 INFO:teuthology.orchestra.run.smithi136.stdout:mgr epoch 9 is available 2024-03-26T04:08:08.372 INFO:teuthology.orchestra.run.smithi136.stdout:Generating a dashboard self-signed certificate... 2024-03-26T04:08:08.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:08 smithi136 ceph-mon[25848]: [26/Mar/2024:04:08:06] ENGINE Bus STARTING 2024-03-26T04:08:08.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:08 smithi136 ceph-mon[25848]: [26/Mar/2024:04:08:06] ENGINE Serving on https://172.21.15.136:7150 2024-03-26T04:08:08.634 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:08 smithi136 ceph-mon[25848]: [26/Mar/2024:04:08:07] ENGINE Serving on http://172.21.15.136:8765 2024-03-26T04:08:08.634 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:08 smithi136 ceph-mon[25848]: [26/Mar/2024:04:08:07] ENGINE Bus STARTED 2024-03-26T04:08:08.634 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:08 smithi136 ceph-mon[25848]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:08:08.634 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:08 smithi136 ceph-mon[25848]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:08:08.634 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:08 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:09.736 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T04:08:09.736 INFO:teuthology.orchestra.run.smithi136.stdout:Creating initial admin user... 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:09 smithi136 ceph-mon[25848]: mgrmap e12: y(active, since 2s) 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:09 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:09 smithi136 ceph-mon[25848]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:09 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:10.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:09 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:10 smithi136 ceph-mon[25848]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:08:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:11.388 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$maU0HPV7mheVpxYJZ/bpo.6Gkxnc8wnKgJPiMMPkmmdjXMk4Okkkq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711426090, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T04:08:11.388 INFO:teuthology.orchestra.run.smithi136.stdout:Fetching dashboard port number... 2024-03-26T04:08:12.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:12 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/2662014633' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T04:08:12.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T04:08:12.982 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:08:12.982 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-03-26T04:08:12.989 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:08:12.990 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T04:08:12.990 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-03-26T04:08:12.990 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T04:08:12.991 INFO:teuthology.orchestra.run.smithi136.stdout:Ceph Dashboard is now available at: 2024-03-26T04:08:12.991 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:12.991 INFO:teuthology.orchestra.run.smithi136.stdout: URL: https://smithi136.front.sepia.ceph.com:8443/ 2024-03-26T04:08:12.991 INFO:teuthology.orchestra.run.smithi136.stdout: User: admin 2024-03-26T04:08:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: Password: 64wtmqioi2 2024-03-26T04:08:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:12.992 INFO:teuthology.orchestra.run.smithi136.stdout:Saving cluster configuration to /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config directory 2024-03-26T04:08:12.992 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling autotune for osd_memory_target 2024-03-26T04:08:14.504 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:14 smithi136 ceph-mon[25848]: mgrmap e13: y(active, since 7s) 2024-03-26T04:08:15.569 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T04:08:15.569 INFO:teuthology.orchestra.run.smithi136.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T04:08:15.569 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:15.569 INFO:teuthology.orchestra.run.smithi136.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:08:15.569 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout: ceph telemetry on 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout:For more information see: 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:15.570 INFO:teuthology.orchestra.run.smithi136.stdout:Bootstrap complete. 2024-03-26T04:08:15.615 INFO:tasks.cephadm:Fetching config... 2024-03-26T04:08:15.615 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:08:15.615 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T04:08:15.631 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T04:08:15.631 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:08:15.631 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T04:08:15.686 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T04:08:15.686 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:08:15.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T04:08:15.752 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T04:08:15.752 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:08:15.752 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T04:08:15.807 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T04:08:15.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq4ec9ZZE2U5j1mWpuJvwHVWDpNRLQwmsmoJNFLM4sVfR+DoyKh3Ys0cPAzKxltgxik2w9TPJiVXwwaHdsXxRSrI6gG+qb9zSAnq4jJmitQb4ZVjXFYraIftseUYY9dxcDyKTGBdAnfev1Dug8+v/JF+QR17sc7uVkanovtEJVVeubbUd3aWaGqib59mjtCw/Fm74R6VM+Y5MTKFwZAKw0Z8e8C7tRLshUaX3YgFD5PxntQYbcanP7Mv0X1VrBjJV5bAYtC9U8sHi/gTLquFRfXFP/1vp7g7Q0cDWr9w0Bzy3hjW3y7q1LCQGKGJYpxYdpdoUAJI3UGqTQhc3GaiB+O9QVPvb3SwfEk/6wZhBWUcxGmnSkLmXX9glH1yvjIfBAwqfWS+v5x0EudQy+VW41NB7kXIq6i3MFR5e862KtdQm1Bw/HZV8bL9nC+OY4EM3bD/ZKWcNsKqeAWsSJ86+cdYilNkyZbLapb65kZMumY525MFekTrYzvOVtGI0Vedc= ceph-276a1e6c-eb26-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:08:15.906 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq4ec9ZZE2U5j1mWpuJvwHVWDpNRLQwmsmoJNFLM4sVfR+DoyKh3Ys0cPAzKxltgxik2w9TPJiVXwwaHdsXxRSrI6gG+qb9zSAnq4jJmitQb4ZVjXFYraIftseUYY9dxcDyKTGBdAnfev1Dug8+v/JF+QR17sc7uVkanovtEJVVeubbUd3aWaGqib59mjtCw/Fm74R6VM+Y5MTKFwZAKw0Z8e8C7tRLshUaX3YgFD5PxntQYbcanP7Mv0X1VrBjJV5bAYtC9U8sHi/gTLquFRfXFP/1vp7g7Q0cDWr9w0Bzy3hjW3y7q1LCQGKGJYpxYdpdoUAJI3UGqTQhc3GaiB+O9QVPvb3SwfEk/6wZhBWUcxGmnSkLmXX9glH1yvjIfBAwqfWS+v5x0EudQy+VW41NB7kXIq6i3MFR5e862KtdQm1Bw/HZV8bL9nC+OY4EM3bD/ZKWcNsKqeAWsSJ86+cdYilNkyZbLapb65kZMumY525MFekTrYzvOVtGI0Vedc= ceph-276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:08:15.922 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq4ec9ZZE2U5j1mWpuJvwHVWDpNRLQwmsmoJNFLM4sVfR+DoyKh3Ys0cPAzKxltgxik2w9TPJiVXwwaHdsXxRSrI6gG+qb9zSAnq4jJmitQb4ZVjXFYraIftseUYY9dxcDyKTGBdAnfev1Dug8+v/JF+QR17sc7uVkanovtEJVVeubbUd3aWaGqib59mjtCw/Fm74R6VM+Y5MTKFwZAKw0Z8e8C7tRLshUaX3YgFD5PxntQYbcanP7Mv0X1VrBjJV5bAYtC9U8sHi/gTLquFRfXFP/1vp7g7Q0cDWr9w0Bzy3hjW3y7q1LCQGKGJYpxYdpdoUAJI3UGqTQhc3GaiB+O9QVPvb3SwfEk/6wZhBWUcxGmnSkLmXX9glH1yvjIfBAwqfWS+v5x0EudQy+VW41NB7kXIq6i3MFR5e862KtdQm1Bw/HZV8bL9nC+OY4EM3bD/ZKWcNsKqeAWsSJ86+cdYilNkyZbLapb65kZMumY525MFekTrYzvOVtGI0Vedc= ceph-276a1e6c-eb26-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:08:15.968 INFO:teuthology.orchestra.run.smithi149.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq4ec9ZZE2U5j1mWpuJvwHVWDpNRLQwmsmoJNFLM4sVfR+DoyKh3Ys0cPAzKxltgxik2w9TPJiVXwwaHdsXxRSrI6gG+qb9zSAnq4jJmitQb4ZVjXFYraIftseUYY9dxcDyKTGBdAnfev1Dug8+v/JF+QR17sc7uVkanovtEJVVeubbUd3aWaGqib59mjtCw/Fm74R6VM+Y5MTKFwZAKw0Z8e8C7tRLshUaX3YgFD5PxntQYbcanP7Mv0X1VrBjJV5bAYtC9U8sHi/gTLquFRfXFP/1vp7g7Q0cDWr9w0Bzy3hjW3y7q1LCQGKGJYpxYdpdoUAJI3UGqTQhc3GaiB+O9QVPvb3SwfEk/6wZhBWUcxGmnSkLmXX9glH1yvjIfBAwqfWS+v5x0EudQy+VW41NB7kXIq6i3MFR5e862KtdQm1Bw/HZV8bL9nC+OY4EM3bD/ZKWcNsKqeAWsSJ86+cdYilNkyZbLapb65kZMumY525MFekTrYzvOVtGI0Vedc= ceph-276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:08:15.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq4ec9ZZE2U5j1mWpuJvwHVWDpNRLQwmsmoJNFLM4sVfR+DoyKh3Ys0cPAzKxltgxik2w9TPJiVXwwaHdsXxRSrI6gG+qb9zSAnq4jJmitQb4ZVjXFYraIftseUYY9dxcDyKTGBdAnfev1Dug8+v/JF+QR17sc7uVkanovtEJVVeubbUd3aWaGqib59mjtCw/Fm74R6VM+Y5MTKFwZAKw0Z8e8C7tRLshUaX3YgFD5PxntQYbcanP7Mv0X1VrBjJV5bAYtC9U8sHi/gTLquFRfXFP/1vp7g7Q0cDWr9w0Bzy3hjW3y7q1LCQGKGJYpxYdpdoUAJI3UGqTQhc3GaiB+O9QVPvb3SwfEk/6wZhBWUcxGmnSkLmXX9glH1yvjIfBAwqfWS+v5x0EudQy+VW41NB7kXIq6i3MFR5e862KtdQm1Bw/HZV8bL9nC+OY4EM3bD/ZKWcNsKqeAWsSJ86+cdYilNkyZbLapb65kZMumY525MFekTrYzvOVtGI0Vedc= ceph-276a1e6c-eb26-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:08:16.028 INFO:teuthology.orchestra.run.smithi204.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq4ec9ZZE2U5j1mWpuJvwHVWDpNRLQwmsmoJNFLM4sVfR+DoyKh3Ys0cPAzKxltgxik2w9TPJiVXwwaHdsXxRSrI6gG+qb9zSAnq4jJmitQb4ZVjXFYraIftseUYY9dxcDyKTGBdAnfev1Dug8+v/JF+QR17sc7uVkanovtEJVVeubbUd3aWaGqib59mjtCw/Fm74R6VM+Y5MTKFwZAKw0Z8e8C7tRLshUaX3YgFD5PxntQYbcanP7Mv0X1VrBjJV5bAYtC9U8sHi/gTLquFRfXFP/1vp7g7Q0cDWr9w0Bzy3hjW3y7q1LCQGKGJYpxYdpdoUAJI3UGqTQhc3GaiB+O9QVPvb3SwfEk/6wZhBWUcxGmnSkLmXX9glH1yvjIfBAwqfWS+v5x0EudQy+VW41NB7kXIq6i3MFR5e862KtdQm1Bw/HZV8bL9nC+OY4EM3bD/ZKWcNsKqeAWsSJ86+cdYilNkyZbLapb65kZMumY525MFekTrYzvOVtGI0Vedc= ceph-276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:08:16.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T04:08:16.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:16 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/2445315137' entity='client.admin' 2024-03-26T04:08:16.286 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:19.685 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T04:08:19.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T04:08:20.362 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:20 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/678842141' entity='client.admin' 2024-03-26T04:08:20.366 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:21.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:21 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:21.653 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:21 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:21.653 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:21 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:08:21.653 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:21 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:21.653 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:21 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:21.653 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:21 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:22.843 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi149 2024-03-26T04:08:22.843 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:08:22.843 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/etc/ceph/ceph.conf 2024-03-26T04:08:22.864 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:08:22.864 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:08:22.920 INFO:tasks.cephadm:Adding host smithi149 to orchestrator... 2024-03-26T04:08:22.921 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch host add smithi149 2024-03-26T04:08:23.158 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:23.379 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:08:23.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:23.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:23.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:23.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:08:23.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T04:08:23.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:08:23.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:08:23.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:23.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:23.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:24.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:24 smithi136 ceph-mon[25848]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:08:25.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:25 smithi136 ceph-mon[25848]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi149", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:08:26.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:26 smithi136 ceph-mon[25848]: Deploying cephadm binary to smithi149 2024-03-26T04:08:26.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:26 smithi136 ceph-mon[25848]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:29.259 INFO:teuthology.orchestra.run.smithi136.stdout:Added host 'smithi149' with addr '172.21.15.149' 2024-03-26T04:08:29.587 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:29 smithi136 ceph-mon[25848]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:29.587 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:29 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:29.587 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:29 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:29.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T04:08:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:30.565 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:30 smithi136 ceph-mon[25848]: Added host smithi149 2024-03-26T04:08:30.565 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:30 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:30.565 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:30 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:31.595 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:31 smithi136 ceph-mon[25848]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:31.815 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:31.816 INFO:teuthology.orchestra.run.smithi136.stdout:[{"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}, {"addr": "172.21.15.149", "hostname": "smithi149", "labels": [], "status": ""}] 2024-03-26T04:08:32.416 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi204 2024-03-26T04:08:32.416 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:08:32.416 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.conf 2024-03-26T04:08:32.435 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:08:32.435 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:08:32.493 INFO:tasks.cephadm:Adding host smithi204 to orchestrator... 2024-03-26T04:08:32.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch host add smithi204 2024-03-26T04:08:32.725 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:33.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:32 smithi136 ceph-mon[25848]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:08:33.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:32 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:33.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:32 smithi136 ceph-mon[25848]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:35.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:35 smithi136 ceph-mon[25848]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:35.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:35 smithi136 ceph-mon[25848]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:08:35.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:35.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:08:36.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:36 smithi136 ceph-mon[25848]: Deploying cephadm binary to smithi204 2024-03-26T04:08:36.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:36 smithi136 ceph-mon[25848]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-26T04:08:36.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:36 smithi136 ceph-mon[25848]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:08:37.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:37 smithi136 ceph-mon[25848]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:38.702 INFO:teuthology.orchestra.run.smithi136.stdout:Added host 'smithi204' with addr '172.21.15.204' 2024-03-26T04:08:39.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T04:08:39.566 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:39.688 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:39 smithi136 ceph-mon[25848]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:08:39.688 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:39 smithi136 ceph-mon[25848]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:08:39.689 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:39 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:39.689 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:39 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:39.689 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:39 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:39.689 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:39 smithi136 ceph-mon[25848]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:39.689 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:39 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:39.689 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:39 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:39.690 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:39 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:40.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:40 smithi136 ceph-mon[25848]: Added host smithi204 2024-03-26T04:08:40.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:40 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:41.440 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:08:41.440 INFO:teuthology.orchestra.run.smithi136.stdout:[{"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}, {"addr": "172.21.15.149", "hostname": "smithi149", "labels": [], "status": ""}, {"addr": "172.21.15.204", "hostname": "smithi204", "labels": [], "status": ""}] 2024-03-26T04:08:41.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:41 smithi136 ceph-mon[25848]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:41.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:42.175 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T04:08:42.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T04:08:42.408 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:42.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:42 smithi136 ceph-mon[25848]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:08:43.559 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:43 smithi136 ceph-mon[25848]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:45.226 INFO:teuthology.orchestra.run.smithi136.stderr:adjusted tunables profile to default 2024-03-26T04:08:45.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:45.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:45.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:45.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:45.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:08:45.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:45.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:08:45.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-26T04:08:45.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1095973120' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:08:45.532 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:45.532 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:08:45.532 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:45.532 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:45.532 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:45.824 INFO:tasks.cephadm:Adding mon.a on smithi136 2024-03-26T04:08:45.824 INFO:tasks.cephadm:Adding mon.b on smithi149 2024-03-26T04:08:45.824 INFO:tasks.cephadm:Adding mon.c on smithi204 2024-03-26T04:08:45.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi136:172.21.15.136=a;smithi149:172.21.15.149=b;smithi204:172.21.15.204=c' 2024-03-26T04:08:46.056 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:08:46.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:46 smithi136 ceph-mon[25848]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:08:46.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:46 smithi136 ceph-mon[25848]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:08:46.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:46 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1095973120' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:08:46.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:46 smithi136 ceph-mon[25848]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:08:47.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:47 smithi136 ceph-mon[25848]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:47.798 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mon update... 2024-03-26T04:08:48.436 DEBUG:teuthology.orchestra.run.smithi149:mon.b> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T04:08:48.439 DEBUG:teuthology.orchestra.run.smithi204:mon.c> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T04:08:48.441 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:08:48.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:08:49.192 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:08:49.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi136:172.21.15.136=a;smithi149:172.21.15.149=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:08:49.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: Saving service mon spec with placement smithi136:172.21.15.136=a;smithi149:172.21.15.149=b;smithi204:172.21.15.204=c;count:3 2024-03-26T04:08:49.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:49.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:49.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:49.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:08:49.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:49.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:08:49.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:49.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: Deploying daemon mon.c on smithi204 2024-03-26T04:08:49.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:48 smithi136 ceph-mon[25848]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:51.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:51 smithi136 ceph-mon[25848]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:52.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T04:08:52.332 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":1,"fsid":"276a1e6c-eb26-11ee-95d0-87774f69a715","modified":"2024-03-26T04:07:07.307842Z","created":"2024-03-26T04:07:07.307842Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T04:08:52.334 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 1 2024-03-26T04:08:52.645 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:52 smithi204 systemd[1]: Starting Ceph mon.c for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:08:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:52 smithi136 ceph-mon[25848]: from='client.? 172.21.15.204:0/2580624812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:08:53.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 podman[30614]: 2024-03-26 04:08:52.95920126 +0000 UTC m=+0.018422236 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:08:53.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 podman[30614]: 2024-03-26 04:08:53.063085615 +0000 UTC m=+0.122306582 container create fce8ab718fac3875670bbf3bbfca54d3ba9583db33de3ac933fd8d9e38e9dc76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 podman[30614]: 2024-03-26 04:08:53.282300318 +0000 UTC m=+0.341521283 container init fce8ab718fac3875670bbf3bbfca54d3ba9583db33de3ac933fd8d9e38e9dc76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124) 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 podman[30614]: 2024-03-26 04:08:53.288051574 +0000 UTC m=+0.347272542 container start fce8ab718fac3875670bbf3bbfca54d3ba9583db33de3ac933fd8d9e38e9dc76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pidfile_write: ignore empty --pid-file 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: load: jerasure load: lrc 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Git sha 0 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: DB SUMMARY 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: DB Session ID: AV6UFUY4WOVUFAYWHWWO 2024-03-26T04:08:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.env: 0x55c226b13ae0 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.info_log: 0x55c228c61920 2024-03-26T04:08:53.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:08:53.751 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.statistics: (nil) 2024-03-26T04:08:53.751 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.use_fsync: 0 2024-03-26T04:08:53.751 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:08:53.751 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:08:53.751 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:08:53.751 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:08:53.751 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.db_log_dir: 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.wal_dir: 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:08:53.752 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.write_buffer_manager: 0x55c228c341e0 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.unordered_write: 0 2024-03-26T04:08:53.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.row_cache: None 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.wal_filter: None 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.wal_compression: 0 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:08:53.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:08:53.755 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_open_files: -1 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Compression algorithms supported: 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:08:53.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: kZSTD supported: 0 2024-03-26T04:08:53.757 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:08:53.757 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:08:53.757 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:08:53.757 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:08:53.757 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:08:53.757 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:08:53.757 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:08:53.757 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:08:53.758 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-26T04:08:53.758 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:08:53.758 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:08:53.758 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.merge_operator: 2024-03-26T04:08:53.758 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_filter: None 2024-03-26T04:08:53.759 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:08:53.759 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:08:53.759 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:08:53.759 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:08:53.759 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c228c61a20) 2024-03-26T04:08:53.759 INFO:journalctl@ceph.mon.c.smithi204.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:08:53.760 INFO:journalctl@ceph.mon.c.smithi204.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:08:53.760 INFO:journalctl@ceph.mon.c.smithi204.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:08:53.760 INFO:journalctl@ceph.mon.c.smithi204.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:08:53.760 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_type: 0 2024-03-26T04:08:53.760 INFO:journalctl@ceph.mon.c.smithi204.stdout: data_block_index_type: 0 2024-03-26T04:08:53.760 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_shortening: 1 2024-03-26T04:08:53.761 INFO:journalctl@ceph.mon.c.smithi204.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:08:53.761 INFO:journalctl@ceph.mon.c.smithi204.stdout: checksum: 4 2024-03-26T04:08:53.761 INFO:journalctl@ceph.mon.c.smithi204.stdout: no_block_cache: 0 2024-03-26T04:08:53.761 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache: 0x55c228c65090 2024-03-26T04:08:53.761 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:08:53.761 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_options: 2024-03-26T04:08:53.761 INFO:journalctl@ceph.mon.c.smithi204.stdout: capacity : 536870912 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: num_shard_bits : 4 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: strict_capacity_limit : 0 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_compressed: (nil) 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: persistent_cache: (nil) 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_size: 4096 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_size_deviation: 10 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_restart_interval: 16 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_block_restart_interval: 1 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: metadata_block_size: 4096 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: partition_filters: 0 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: use_delta_encoding: 1 2024-03-26T04:08:53.762 INFO:journalctl@ceph.mon.c.smithi204.stdout: filter_policy: bloomfilter 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout: whole_key_filtering: 1 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout: verify_compression: 0 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout: format_version: 5 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout: enable_index_compression: 1 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_align: 0 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout: max_auto_readahead_size: 262144 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout: prepopulate_block_cache: 0 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:08:53.763 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compression: NoCompression 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.num_levels: 7 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:08:53.764 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:08:53.765 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:08:53.766 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:08:53.767 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:08:53.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.ttl: 2592000 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:08:53.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 93a34393-13c6-46b6-88d7-49bee2cd8853 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426133317547, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426133317970, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711426133, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "93a34393-13c6-46b6-88d7-49bee2cd8853", "db_session_id": "AV6UFUY4WOVUFAYWHWWO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426133318051, "job": 1, "event": "recovery_finished"} 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c228d56000 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: DB pointer 0x55c228d40000 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:08:53.770 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** DB Stats ** 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** Compaction Stats [default] ** 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:08:53.771 INFO:journalctl@ceph.mon.c.smithi204.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** Compaction Stats [default] ** 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:08:53.772 INFO:journalctl@ceph.mon.c.smithi204.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: Block cache BinnedLRUCache@0x55c228c65090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:08:53.773 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: using public_addr v2:172.21.15.204:0/0 -> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: starting mon.c rank -1 at public addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] at bind addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(???) e0 preinit fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).mds e1 new map 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout: e1 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout: legacy client fscid: -1 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout: No filesystems configured 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T04:08:53.774 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:08:53.775 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Deploying cephadm binary to smithi204 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Added host smithi204 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.777 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/1095973120' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:08:53.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/1095973120' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi136:172.21.15.136=a;smithi149:172.21.15.149=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Saving service mon spec with placement smithi136:172.21.15.136=a;smithi149:172.21.15.149=b;smithi204:172.21.15.204=c;count:3 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: Deploying daemon mon.c on smithi204 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:53.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:53.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: from='client.? 172.21.15.204:0/2580624812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:08:53.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 ceph-mon[30652]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T04:08:53.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 bash[30614]: fce8ab718fac3875670bbf3bbfca54d3ba9583db33de3ac933fd8d9e38e9dc76 2024-03-26T04:08:53.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:53 smithi204 systemd[1]: Started Ceph mon.c for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:08:53.882 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:08:53.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:08:54.115 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: mon.a calling monitor election 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: mon.c calling monitor election 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:08:58.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: monmap e2: 2 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:08:58.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: fsmap 2024-03-26T04:08:58.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:08:58.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: mgrmap e13: y(active, since 51s) 2024-03-26T04:08:58.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: overall HEALTH_OK 2024-03-26T04:08:58.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:58.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:58 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:58.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: mon.a calling monitor election 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: mon.c calling monitor election 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: monmap e2: 2 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: fsmap 2024-03-26T04:08:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:08:58.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: mgrmap e13: y(active, since 51s) 2024-03-26T04:08:58.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: overall HEALTH_OK 2024-03-26T04:08:58.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:58.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:58 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:58.965 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T04:08:58.965 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"276a1e6c-eb26-11ee-95d0-87774f69a715","modified":"2024-03-26T04:08:53.340443Z","created":"2024-03-26T04:07:07.307842Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T04:08:58.967 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-03-26T04:08:59.543 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:59 smithi204 ceph-mon[30652]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:59.543 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:59 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:59.543 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:59 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:08:59.543 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:59 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:59.543 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:59 smithi204 ceph-mon[30652]: Deploying daemon mon.b on smithi149 2024-03-26T04:08:59.544 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:59 smithi204 ceph-mon[30652]: from='client.? 172.21.15.204:0/3534047978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:08:59.544 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:08:59 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:08:59.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:59 smithi136 ceph-mon[25848]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:08:59.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:59 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:08:59.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:59 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:08:59.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:59 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:59.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:59 smithi136 ceph-mon[25848]: Deploying daemon mon.b on smithi149 2024-03-26T04:08:59.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:59 smithi136 ceph-mon[25848]: from='client.? 172.21.15.204:0/3534047978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:08:59.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:08:59 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:00.569 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:09:00.569 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:09:00.813 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:09:01.615 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:01 smithi204 ceph-mon[30652]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:01.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:01 smithi136 ceph-mon[25848]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.067 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:01 smithi149 podman[30619]: 2024-03-26 04:09:01.876261579 +0000 UTC m=+0.018560579 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:09:02.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 podman[30619]: 2024-03-26 04:09:02.179136443 +0000 UTC m=+0.321435428 container init a2d968c1e135db7c2e367062433b6d81fffeb5e3201fc1fc75b135b37f2739ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-b, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-26T04:09:02.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 podman[30619]: 2024-03-26 04:09:02.184768117 +0000 UTC m=+0.327067104 container start a2d968c1e135db7c2e367062433b6d81fffeb5e3201fc1fc75b135b37f2739ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-5e70124, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T04:09:02.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:09:02.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T04:09:02.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pidfile_write: ignore empty --pid-file 2024-03-26T04:09:02.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: load: jerasure load: lrc 2024-03-26T04:09:02.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:09:02.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Git sha 0 2024-03-26T04:09:02.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:09:02.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: DB SUMMARY 2024-03-26T04:09:02.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: DB Session ID: 35BZWA99E1388EP4OIAO 2024-03-26T04:09:02.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:09:02.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.env: 0x55cf63a2fae0 2024-03-26T04:09:02.491 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:09:02.491 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.info_log: 0x55cf64f99920 2024-03-26T04:09:02.491 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:09:02.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.statistics: (nil) 2024-03-26T04:09:02.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.use_fsync: 0 2024-03-26T04:09:02.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:09:02.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:09:02.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:09:02.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:09:02.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:09:02.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:09:02.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.db_log_dir: 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.wal_dir: 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:09:02.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.write_buffer_manager: 0x55cf64f6c1e0 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.unordered_write: 0 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:09:02.497 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.row_cache: None 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.wal_filter: None 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.wal_compression: 0 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:09:02.498 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:09:02.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_open_files: -1 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Compression algorithms supported: 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: kZSTD supported: 0 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:09:02.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.merge_operator: 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_filter: None 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cf64f99a20) 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:09:02.501 INFO:journalctl@ceph.mon.b.smithi149.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_type: 0 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: data_block_index_type: 0 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_shortening: 1 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: checksum: 4 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: no_block_cache: 0 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache: 0x55cf64f9d090 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_options: 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: capacity : 536870912 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: num_shard_bits : 4 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: strict_capacity_limit : 0 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:09:02.502 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_compressed: (nil) 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: persistent_cache: (nil) 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_size: 4096 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_size_deviation: 10 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_restart_interval: 16 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_block_restart_interval: 1 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: metadata_block_size: 4096 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: partition_filters: 0 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: use_delta_encoding: 1 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: filter_policy: bloomfilter 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: whole_key_filtering: 1 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: verify_compression: 0 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: format_version: 5 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: enable_index_compression: 1 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_align: 0 2024-03-26T04:09:02.503 INFO:journalctl@ceph.mon.b.smithi149.stdout: max_auto_readahead_size: 262144 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout: prepopulate_block_cache: 0 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compression: NoCompression 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.num_levels: 7 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:09:02.504 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:09:02.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:09:02.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:09:02.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:09:02.508 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.ttl: 2592000 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:09:02.509 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 739c5596-5e92-46e4-8fe0-c8c7804f4a7f 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426142214412, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426142214845, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711426142, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "739c5596-5e92-46e4-8fe0-c8c7804f4a7f", "db_session_id": "35BZWA99E1388EP4OIAO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426142214925, "job": 1, "event": "recovery_finished"} 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:09:02.510 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cf6508e000 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: DB pointer 0x55cf65078000 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** DB Stats ** 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2024-03-26T04:09:02.511 INFO:journalctl@ceph.mon.b.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:09:02.512 INFO:journalctl@ceph.mon.b.smithi149.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: Block cache BinnedLRUCache@0x55cf64f9d090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: using public_addr v2:172.21.15.149:0/0 -> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: starting mon.b rank -1 at public addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] at bind addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:09:02.513 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(???) e0 preinit fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout: e1 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout: legacy client fscid: -1 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout: No filesystems configured 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T04:09:02.514 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.515 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Deploying cephadm binary to smithi204 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.516 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Added host smithi204 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:02.517 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/1095973120' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/1095973120' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi136:172.21.15.136=a;smithi149:172.21.15.149=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:02.518 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Saving service mon spec with placement smithi136:172.21.15.136=a;smithi149:172.21.15.149=b;smithi204:172.21.15.204=c;count:3 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Deploying daemon mon.c on smithi204 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='client.? 172.21.15.204:0/2580624812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:09:02.519 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.a calling monitor election 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.c calling monitor election 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: monmap e2: 2 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: fsmap 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:09:02.520 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mgrmap e13: y(active, since 51s) 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: overall HEALTH_OK 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: Deploying daemon mon.b on smithi149 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='client.? 172.21.15.204:0/3534047978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 ceph-mon[30657]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 bash[30619]: a2d968c1e135db7c2e367062433b6d81fffeb5e3201fc1fc75b135b37f2739ad 2024-03-26T04:09:02.521 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:02 smithi149 systemd[1]: Started Ceph mon.b for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:09:07.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: mon.c calling monitor election 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: mon.a calling monitor election 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:07.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: fsmap 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: mgrmap e13: y(active, since 60s) 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:07.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:07.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:09:07.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: mon.c calling monitor election 2024-03-26T04:09:07.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: mon.a calling monitor election 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: fsmap 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:09:07.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: mgrmap e13: y(active, since 60s) 2024-03-26T04:09:07.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:09:07.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:09:07.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:09:07.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2024-03-26T04:09:07.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:07.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:07.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:07.749 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-26T04:09:08.660 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":3,"fsid":"276a1e6c-eb26-11ee-95d0-87774f69a715","modified":"2024-03-26T04:09:02.240664Z","created":"2024-03-26T04:07:07.307842Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T04:09:08.660 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:08 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:08.660 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:08 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:08.662 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 3 2024-03-26T04:09:08.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:08 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:08.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:08 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:09.487 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T04:09:09.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T04:09:09.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:09:09.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: mon.c calling monitor election 2024-03-26T04:09:09.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: mon.a calling monitor election 2024-03-26T04:09:09.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:09.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:09:09.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:09.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: fsmap 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: mgrmap e13: y(active, since 60s) 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:09:09.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2024-03-26T04:09:09.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:09.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:09.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:09.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:09.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:09.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:09.740 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:09:10.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: mon.b calling monitor election 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: mon.b calling monitor election 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: mon.c calling monitor election 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: mon.a calling monitor election 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: fsmap 2024-03-26T04:09:10.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: mgrmap e13: y(active, since 62s) 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: Cluster is now healthy 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: overall HEALTH_OK 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:09:10.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:10.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:09:10.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:10.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: Reconfiguring daemon mon.a on smithi136 2024-03-26T04:09:10.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: mon.b calling monitor election 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: mon.b calling monitor election 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: mon.c calling monitor election 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: mon.a calling monitor election 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:09:10.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: fsmap 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: mgrmap e13: y(active, since 62s) 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: Cluster is now healthy 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: overall HEALTH_OK 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:09:10.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:10.500 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:09:10.500 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:10.500 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: Reconfiguring daemon mon.a on smithi136 2024-03-26T04:09:10.500 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:10.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: mon.b calling monitor election 2024-03-26T04:09:10.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T04:09:10.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-26T04:09:10.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-26T04:09:10.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: mon.b calling monitor election 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: mon.c calling monitor election 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: mon.a calling monitor election 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: fsmap 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: mgrmap e13: y(active, since 62s) 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: Cluster is now healthy 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: overall HEALTH_OK 2024-03-26T04:09:10.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: Reconfiguring daemon mon.a on smithi136 2024-03-26T04:09:10.740 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:09:11.513 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:11 smithi136 ceph-mon[25848]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:11.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:11 smithi149 ceph-mon[30657]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:11.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:11 smithi204 ceph-mon[30652]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:12.701 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:12.701 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:12.701 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:12.701 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:12.701 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:09:12.702 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:12.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:12.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:12.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:12.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:12.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:09:12.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:12.776 INFO:teuthology.orchestra.run.smithi136.stdout:# minimal ceph.conf for 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:09:12.776 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-03-26T04:09:12.776 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:09:12.776 INFO:teuthology.orchestra.run.smithi136.stdout: mon_host = [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-03-26T04:09:12.811 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:12.811 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:12.811 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:12.811 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:12.811 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:09:12.811 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:13.394 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T04:09:13.395 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:09:13.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:09:13.426 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:09:13.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:09:13.495 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:09:13.495 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:09:13.525 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:09:13.525 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:09:13.596 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:09:13.596 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:09:13.631 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:09:13.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:09:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:13 smithi136 ceph-mon[25848]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:09:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:13 smithi136 ceph-mon[25848]: Reconfiguring daemon mon.b on smithi149 2024-03-26T04:09:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:13 smithi136 ceph-mon[25848]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:13 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/3066143444' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:13.700 INFO:tasks.cephadm:Adding mgr.y on smithi136 2024-03-26T04:09:13.700 INFO:tasks.cephadm:Adding mgr.x on smithi149 2024-03-26T04:09:13.700 INFO:tasks.cephadm:Adding mgr.z on smithi204 2024-03-26T04:09:13.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi136=y;smithi149=x;smithi204=z' 2024-03-26T04:09:13.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:13 smithi149 ceph-mon[30657]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:09:13.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:13 smithi149 ceph-mon[30657]: Reconfiguring daemon mon.b on smithi149 2024-03-26T04:09:13.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:13 smithi149 ceph-mon[30657]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:13.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:13 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/3066143444' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:13.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:13 smithi204 ceph-mon[30652]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:09:13.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:13 smithi204 ceph-mon[30652]: Reconfiguring daemon mon.b on smithi149 2024-03-26T04:09:13.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:13 smithi204 ceph-mon[30652]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:13.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:13 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/3066143444' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:13.977 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:09:14.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:14 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:14.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:14 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:14.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:14 smithi204 ceph-mon[30652]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:09:14.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:14 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:14.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:14 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:09:14.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:14 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:14.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:14 smithi204 ceph-mon[30652]: Reconfiguring daemon mon.c on smithi204 2024-03-26T04:09:14.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:14 smithi204 ceph-mon[30652]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:14.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:14 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:14.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:14 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:14.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:14 smithi136 ceph-mon[25848]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:09:14.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:14 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:14.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:14 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:09:14.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:14 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:14.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:14 smithi136 ceph-mon[25848]: Reconfiguring daemon mon.c on smithi204 2024-03-26T04:09:14.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:14 smithi136 ceph-mon[25848]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:14 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:14 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:14 smithi149 ceph-mon[30657]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:09:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:14 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:09:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:14 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:09:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:14 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:14 smithi149 ceph-mon[30657]: Reconfiguring daemon mon.c on smithi204 2024-03-26T04:09:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:14 smithi149 ceph-mon[30657]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:16.384 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mgr update... 2024-03-26T04:09:17.211 DEBUG:teuthology.orchestra.run.smithi149:mgr.x> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T04:09:17.214 DEBUG:teuthology.orchestra.run.smithi204:mgr.z> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.z.service 2024-03-26T04:09:17.254 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T04:09:17.254 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:09:17.254 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:09:17.269 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:09:17.270 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/nvme1n1 2024-03-26T04:09:17.325 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/nvme1n1 2024-03-26T04:09:17.325 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:17.325 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-26T04:09:17.325 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:17.325 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:17.325 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 04:08:19.406088524 +0000 2024-03-26T04:09:17.325 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 04:05:55.760227556 +0000 2024-03-26T04:09:17.325 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 04:05:55.760227556 +0000 2024-03-26T04:09:17.325 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 04:05:55.751227758 +0000 2024-03-26T04:09:17.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T04:09:17.391 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T04:09:17.391 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T04:09:17.391 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000215047 s, 2.4 MB/s 2024-03-26T04:09:17.392 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T04:09:17.448 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/nvme2n1 2024-03-26T04:09:17.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi136=y;smithi149=x;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:17.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: Saving service mgr spec with placement smithi136=y;smithi149=x;smithi204=z;count:3 2024-03-26T04:09:17.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:17.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:17.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:17.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:17.469 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.469 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:09:17.469 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:09:17.469 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:09:17.469 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:17 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/nvme2n1 2024-03-26T04:09:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:17.504 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-26T04:09:17.504 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:17.504 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:17.504 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 04:08:19.416088301 +0000 2024-03-26T04:09:17.504 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 04:05:55.913224130 +0000 2024-03-26T04:09:17.504 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 04:05:55.913224130 +0000 2024-03-26T04:09:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 04:05:55.899224443 +0000 2024-03-26T04:09:17.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T04:09:17.569 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T04:09:17.570 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T04:09:17.570 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000219639 s, 2.3 MB/s 2024-03-26T04:09:17.571 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T04:09:17.626 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/nvme3n1 2024-03-26T04:09:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/nvme3n1 2024-03-26T04:09:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:17.681 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-26T04:09:17.681 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:17.681 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:17.682 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 04:08:19.427088055 +0000 2024-03-26T04:09:17.682 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 04:05:56.062229041 +0000 2024-03-26T04:09:17.682 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 04:05:56.062229041 +0000 2024-03-26T04:09:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 04:05:56.053228550 +0000 2024-03-26T04:09:17.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T04:09:17.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi136=y;smithi149=x;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:17.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: Saving service mgr spec with placement smithi136=y;smithi149=x;smithi204=z;count:3 2024-03-26T04:09:17.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:17.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:17.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:17.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:17.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:09:17.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:09:17.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:09:17.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:17 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:17.723 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T04:09:17.724 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T04:09:17.724 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000202 s, 2.5 MB/s 2024-03-26T04:09:17.725 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T04:09:17.737 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi136=y;smithi149=x;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:17.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: Saving service mgr spec with placement smithi136=y;smithi149=x;smithi204=z;count:3 2024-03-26T04:09:17.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:17.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:17.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:17.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:17.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:17.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:09:17.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:09:17.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:09:17.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:17 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:17.780 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/nvme4n1 2024-03-26T04:09:17.835 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/nvme4n1 2024-03-26T04:09:17.835 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:17.836 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-26T04:09:17.836 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:17.836 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:17.836 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 04:08:19.438087810 +0000 2024-03-26T04:09:17.836 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 04:05:56.215237383 +0000 2024-03-26T04:09:17.836 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 04:05:56.215237383 +0000 2024-03-26T04:09:17.836 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 04:05:56.203236729 +0000 2024-03-26T04:09:17.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T04:09:17.901 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T04:09:17.901 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T04:09:17.901 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000251196 s, 2.0 MB/s 2024-03-26T04:09:17.902 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T04:09:17.958 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:09:17.959 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:09:17.978 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:09:17.978 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/nvme1n1 2024-03-26T04:09:18.035 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/nvme1n1 2024-03-26T04:09:18.035 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:18.035 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-26T04:09:18.035 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:18.035 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:18.036 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-26 04:08:34.146884106 +0000 2024-03-26T04:09:18.036 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-26 04:05:57.392880387 +0000 2024-03-26T04:09:18.036 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-26 04:05:57.392880387 +0000 2024-03-26T04:09:18.036 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-26 04:05:57.380880616 +0000 2024-03-26T04:09:18.036 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T04:09:18.105 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-26T04:09:18.105 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-26T04:09:18.105 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000328595 s, 1.6 MB/s 2024-03-26T04:09:18.106 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T04:09:18.164 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/nvme2n1 2024-03-26T04:09:18.221 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/nvme2n1 2024-03-26T04:09:18.221 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:18.221 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-26T04:09:18.221 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:18.221 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:18.221 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-26 04:08:34.157883896 +0000 2024-03-26T04:09:18.222 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-26 04:05:57.541877540 +0000 2024-03-26T04:09:18.222 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-26 04:05:57.541877540 +0000 2024-03-26T04:09:18.222 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-26 04:05:57.533877693 +0000 2024-03-26T04:09:18.222 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T04:09:18.290 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-26T04:09:18.290 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-26T04:09:18.290 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000276905 s, 1.8 MB/s 2024-03-26T04:09:18.291 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T04:09:18.349 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/nvme3n1 2024-03-26T04:09:18.404 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/nvme3n1 2024-03-26T04:09:18.405 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:18.405 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-26T04:09:18.405 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:18.405 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:18.405 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-26 04:08:34.168883686 +0000 2024-03-26T04:09:18.405 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-26 04:05:57.688874731 +0000 2024-03-26T04:09:18.405 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-26 04:05:57.688874731 +0000 2024-03-26T04:09:18.405 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-26 04:05:57.677874941 +0000 2024-03-26T04:09:18.405 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T04:09:18.472 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-26T04:09:18.472 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-26T04:09:18.473 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000243848 s, 2.1 MB/s 2024-03-26T04:09:18.474 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T04:09:18.529 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/nvme4n1 2024-03-26T04:09:18.584 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/nvme4n1 2024-03-26T04:09:18.585 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:18.585 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-26T04:09:18.585 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:18.585 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:18.585 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-03-26 04:08:34.179883476 +0000 2024-03-26T04:09:18.585 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-03-26 04:05:57.836871903 +0000 2024-03-26T04:09:18.585 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-03-26 04:05:57.836871903 +0000 2024-03-26T04:09:18.585 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-03-26 04:05:57.823872151 +0000 2024-03-26T04:09:18.585 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T04:09:18.649 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:18 smithi149 ceph-mon[30657]: Deploying daemon mgr.z on smithi204 2024-03-26T04:09:18.651 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-03-26T04:09:18.651 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-03-26T04:09:18.651 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000245973 s, 2.1 MB/s 2024-03-26T04:09:18.653 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T04:09:18.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:18 smithi136 ceph-mon[25848]: Deploying daemon mgr.z on smithi204 2024-03-26T04:09:18.708 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:09:18.708 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:09:18.725 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T04:09:18.725 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/nvme1n1 2024-03-26T04:09:18.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:18 smithi204 ceph-mon[30652]: Deploying daemon mgr.z on smithi204 2024-03-26T04:09:18.761 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/nvme1n1 2024-03-26T04:09:18.761 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:18.761 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-26T04:09:18.761 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:18.761 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:18.761 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-26 04:08:43.134800092 +0000 2024-03-26T04:09:18.761 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-26 04:05:59.039116633 +0000 2024-03-26T04:09:18.761 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-26 04:05:59.039116633 +0000 2024-03-26T04:09:18.761 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-26 04:05:59.027116803 +0000 2024-03-26T04:09:18.762 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T04:09:18.829 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-26T04:09:18.829 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-26T04:09:18.829 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000344006 s, 1.5 MB/s 2024-03-26T04:09:18.830 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T04:09:18.887 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/nvme2n1 2024-03-26T04:09:18.943 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/nvme2n1 2024-03-26T04:09:18.943 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:18.943 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-26T04:09:18.943 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:18.943 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:18.943 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-26 04:08:43.144799952 +0000 2024-03-26T04:09:18.943 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-26 04:05:59.187114535 +0000 2024-03-26T04:09:18.943 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-26 04:05:59.187114535 +0000 2024-03-26T04:09:18.943 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-26 04:05:59.175114705 +0000 2024-03-26T04:09:18.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T04:09:19.069 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-26T04:09:19.069 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-26T04:09:19.069 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000323363 s, 1.6 MB/s 2024-03-26T04:09:19.071 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T04:09:19.086 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/nvme3n1 2024-03-26T04:09:19.145 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/nvme3n1 2024-03-26T04:09:19.145 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:19.145 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-26T04:09:19.145 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:19.145 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:19.145 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-26 04:08:43.155799799 +0000 2024-03-26T04:09:19.145 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-26 04:05:59.334112452 +0000 2024-03-26T04:09:19.145 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-26 04:05:59.334112452 +0000 2024-03-26T04:09:19.145 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-26 04:05:59.323112608 +0000 2024-03-26T04:09:19.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T04:09:19.286 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-26T04:09:19.286 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-26T04:09:19.286 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000235838 s, 2.2 MB/s 2024-03-26T04:09:19.287 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T04:09:19.302 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/nvme4n1 2024-03-26T04:09:19.358 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/nvme4n1 2024-03-26T04:09:19.358 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T04:09:19.358 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-26T04:09:19.358 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T04:09:19.358 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T04:09:19.358 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-26 04:08:43.165799659 +0000 2024-03-26T04:09:19.358 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-26 04:05:59.482110355 +0000 2024-03-26T04:09:19.358 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-26 04:05:59.482110355 +0000 2024-03-26T04:09:19.359 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-26 04:05:59.471110511 +0000 2024-03-26T04:09:19.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T04:09:19.401 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:19 smithi204 systemd[1]: Starting Ceph mgr.z for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:09:19.425 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-26T04:09:19.425 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-26T04:09:19.425 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000207866 s, 2.5 MB/s 2024-03-26T04:09:19.426 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T04:09:19.482 INFO:tasks.cephadm:Deploying osd.0 on smithi136 with /dev/nvme4n1... 2024-03-26T04:09:19.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T04:09:19.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:19 smithi136 ceph-mon[25848]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:19.715 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:19 smithi204 ceph-mon[30652]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:19.715 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:19 smithi204 podman[32685]: 2024-03-26 04:09:19.505344766 +0000 UTC m=+0.098334184 container create 857705d92f9366a52fa0593df7950e096ce4f70d269b9f03c0743e8d9bd3fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T04:09:19.716 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:19 smithi204 podman[32685]: 2024-03-26 04:09:19.425062755 +0000 UTC m=+0.018052175 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:09:19.716 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:19 smithi204 podman[32685]: 2024-03-26 04:09:19.716186004 +0000 UTC m=+0.309175437 container init 857705d92f9366a52fa0593df7950e096ce4f70d269b9f03c0743e8d9bd3fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T04:09:19.721 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:09:19.737 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:19 smithi149 ceph-mon[30657]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:19.997 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:19 smithi204 podman[32685]: 2024-03-26 04:09:19.722431478 +0000 UTC m=+0.315420897 container start 857705d92f9366a52fa0593df7950e096ce4f70d269b9f03c0743e8d9bd3fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T04:09:19.997 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:19 smithi204 ceph-mgr[32759]: -- 172.21.15.204:0/3939437019 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5653a0d301a0 con 0x5653a0d07000 2024-03-26T04:09:19.998 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:19 smithi204 bash[32685]: 857705d92f9366a52fa0593df7950e096ce4f70d269b9f03c0743e8d9bd3fb40 2024-03-26T04:09:19.998 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:19 smithi204 systemd[1]: Started Ceph mgr.z for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:09:20.747 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:20.261+0000 7f4a380f6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:09:20.747 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:20.353+0000 7f4a380f6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:09:21.094 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:20 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:21.094 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:20 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:21.094 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:20 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:21.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:20 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:09:21.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:20 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:09:21.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:20 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:09:21.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:20 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:21.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:20 smithi136 ceph-mon[25848]: Deploying daemon mgr.x on smithi149 2024-03-26T04:09:21.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:20 smithi136 ceph-mon[25848]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:21.236 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:20.993+0000 7f4a380f6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:09:21.236 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:21 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:21.086+0000 7f4a380f6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:09:21.236 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:21 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:21.235+0000 7f4a380f6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:09:21.237 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:21.237 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:21.237 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:21.237 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:09:21.237 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:09:21.237 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:09:21.237 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:21.237 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-mon[30652]: Deploying daemon mgr.x on smithi149 2024-03-26T04:09:21.237 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:20 smithi204 ceph-mon[30652]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:21.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:20 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:20 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:20 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:20 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:09:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:20 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:09:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:20 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:09:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:20 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:20 smithi149 ceph-mon[30657]: Deploying daemon mgr.x on smithi149 2024-03-26T04:09:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:20 smithi149 ceph-mon[30657]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:21.497 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:21 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:21.332+0000 7f4a380f6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:09:21.497 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:21 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:21.414+0000 7f4a380f6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:09:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T04:09:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T04:09:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T04:09:21.645 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-03-26T04:09:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-03-26T04:09:21.645 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.010446 s, 1.0 GB/s 2024-03-26T04:09:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-03-26T04:09:21.977 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:21 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:21.687+0000 7f4a380f6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:09:21.977 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:21 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:21.796+0000 7f4a380f6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:09:21.978 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:21 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:21.882+0000 7f4a380f6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:09:21.978 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:21 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:21.977+0000 7f4a380f6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:09:22.247 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:22 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:22.153+0000 7f4a380f6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:09:22.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi136:/dev/nvme4n1 2024-03-26T04:09:22.341 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:22 smithi149 podman[32099]: 2024-03-26 04:09:22.126373053 +0000 UTC m=+0.018158319 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:09:22.502 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:09:22.738 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:22 smithi149 podman[32099]: 2024-03-26 04:09:22.423437357 +0000 UTC m=+0.315222612 container init f6579a30c564cbd589092cde15eb49e33734ab4743603b097c1bd1e711304be7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-26T04:09:22.738 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:22 smithi149 podman[32099]: 2024-03-26 04:09:22.429028937 +0000 UTC m=+0.320814202 container start f6579a30c564cbd589092cde15eb49e33734ab4743603b097c1bd1e711304be7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-26T04:09:22.738 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:22 smithi149 ceph-mgr[32137]: -- 172.21.15.149:0/3697565875 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5627ba72a1a0 con 0x5627ba701000 2024-03-26T04:09:22.738 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:22 smithi149 bash[32099]: f6579a30c564cbd589092cde15eb49e33734ab4743603b097c1bd1e711304be7 2024-03-26T04:09:22.738 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:22 smithi149 systemd[1]: Started Ceph mgr.x for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:09:23.488 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:22 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:22.993+0000 7feac11dc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:09:23.488 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:23 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:23.090+0000 7feac11dc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:09:23.497 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:23.142+0000 7f4a380f6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:09:23.497 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:23.286+0000 7f4a380f6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:09:23.620 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:23 smithi136 ceph-mon[25848]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:23.620 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.620 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.620 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:23.899 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:23 smithi149 ceph-mon[30657]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:23.899 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:23 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.899 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:23 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.899 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:23 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.899 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:23 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.899 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:23 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:23.900 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:23 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:23.802+0000 7feac11dc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:09:23.900 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:23 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:23.897+0000 7feac11dc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:09:23.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-mon[30652]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:23.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:23.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:23.937 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:23.641+0000 7f4a380f6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:09:23.937 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:23.741+0000 7f4a380f6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:09:23.938 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:23 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:23.936+0000 7f4a380f6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:09:24.167 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:24 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:24.055+0000 7feac11dc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:09:24.167 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:24 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:24.165+0000 7feac11dc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:09:24.247 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:24 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:24.129+0000 7f4a380f6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:09:24.487 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:24 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:24.250+0000 7feac11dc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:09:24.682 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:24 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:24.413+0000 7f4a380f6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:09:24.682 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:24 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:24.681+0000 7f4a380f6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:09:24.811 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:24 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:24.517+0000 7feac11dc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:09:24.811 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:24 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:24.627+0000 7feac11dc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:09:24.811 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:24 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:24.714+0000 7feac11dc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:09:24.811 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:24 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:24.809+0000 7feac11dc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:09:24.997 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:24 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:24.916+0000 7f4a380f6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:09:25.102 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:24 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:24.988+0000 7feac11dc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:09:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:09:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:09:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:09:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: Reconfiguring daemon mgr.y on smithi136 2024-03-26T04:09:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:09:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:09:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:25 smithi136 ceph-mon[25848]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:25.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: Reconfiguring daemon mgr.y on smithi136 2024-03-26T04:09:25.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:25.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:09:25.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:09:25.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:25.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-mon[30657]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:25.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: Reconfiguring daemon mgr.y on smithi136 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:25.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-mon[30652]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:25.997 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:25 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:25.722+0000 7f4a380f6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:09:26.238 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:25 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:25.967+0000 7feac11dc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:09:26.238 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:26 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:26.098+0000 7feac11dc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:09:26.606 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:26 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:26.297+0000 7f4a380f6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:09:26.606 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:26 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:26.392+0000 7f4a380f6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:09:26.607 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:09:26 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:09:26.605+0000 7f4a380f6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:09:26.719 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:26 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:26.439+0000 7feac11dc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:09:26.719 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:26 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:26.526+0000 7feac11dc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:09:26.719 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:26 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:26.718+0000 7feac11dc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:09:26.987 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:26 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:26.899+0000 7feac11dc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:09:27.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:27.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:27.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:27.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:27.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: Standby manager daemon z started 2024-03-26T04:09:27.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:09:27.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:09:27.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:09:27.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:27 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:09:27.447 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:27.180+0000 7feac11dc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:09:27.447 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:27.445+0000 7feac11dc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:09:27.447 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.447 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.447 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:27.447 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:27.447 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:27.447 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.448 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:27.448 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.448 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: Standby manager daemon z started 2024-03-26T04:09:27.448 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:09:27.448 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:09:27.448 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:09:27.448 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:09:27.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: Standby manager daemon z started 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:09:27.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:27 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.204:0/1592832149' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:09:27.737 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:27 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:27.683+0000 7feac11dc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:09:28.737 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:28 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:28.485+0000 7feac11dc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:09:28.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:28 smithi149 ceph-mon[30657]: mgrmap e14: y(active, since 81s), standbys: z 2024-03-26T04:09:28.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:28 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:09:28.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:28 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/1783459947' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bc9716b-db6c-4d3a-a77b-0424e38088a9"}]: dispatch 2024-03-26T04:09:28.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:28 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/1783459947' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bc9716b-db6c-4d3a-a77b-0424e38088a9"}]': finished 2024-03-26T04:09:28.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:28 smithi149 ceph-mon[30657]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:09:28.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:28 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:28.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:28 smithi204 ceph-mon[30652]: mgrmap e14: y(active, since 81s), standbys: z 2024-03-26T04:09:28.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:28 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:09:28.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:28 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/1783459947' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bc9716b-db6c-4d3a-a77b-0424e38088a9"}]: dispatch 2024-03-26T04:09:28.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:28 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/1783459947' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bc9716b-db6c-4d3a-a77b-0424e38088a9"}]': finished 2024-03-26T04:09:28.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:28 smithi204 ceph-mon[30652]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:09:28.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:28 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:28.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:28 smithi136 ceph-mon[25848]: mgrmap e14: y(active, since 81s), standbys: z 2024-03-26T04:09:28.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:28 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:09:28.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:28 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1783459947' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bc9716b-db6c-4d3a-a77b-0424e38088a9"}]: dispatch 2024-03-26T04:09:28.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:28 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1783459947' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bc9716b-db6c-4d3a-a77b-0424e38088a9"}]': finished 2024-03-26T04:09:28.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:28 smithi136 ceph-mon[25848]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:09:28.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:28 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:29.375 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:29 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:29.066+0000 7feac11dc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:09:29.375 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:29 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:29.163+0000 7feac11dc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:09:29.375 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:09:29 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:09:29.374+0000 7feac11dc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:09:29.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:29 smithi149 ceph-mon[30657]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:29.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:29 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/2220534866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:09:29.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:29 smithi149 ceph-mon[30657]: Standby manager daemon x started 2024-03-26T04:09:29.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:29 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:09:29.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:29 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:09:29.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:29 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:09:29.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:29 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:09:29.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:29 smithi136 ceph-mon[25848]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:29.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:29 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/2220534866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:09:29.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:29 smithi136 ceph-mon[25848]: Standby manager daemon x started 2024-03-26T04:09:29.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:29 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:09:29.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:29 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:09:29.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:29 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:09:29.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:29 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:09:29.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:29 smithi204 ceph-mon[30652]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:29.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:29 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/2220534866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:09:29.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:29 smithi204 ceph-mon[30652]: Standby manager daemon x started 2024-03-26T04:09:29.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:29 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:09:29.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:29 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:09:29.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:29 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:09:29.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:29 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.149:0/2254683874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:09:31.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:31 smithi136 ceph-mon[25848]: mgrmap e15: y(active, since 83s), standbys: z, x 2024-03-26T04:09:31.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:31 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:09:31.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:31 smithi136 ceph-mon[25848]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:31.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:31 smithi149 ceph-mon[30657]: mgrmap e15: y(active, since 83s), standbys: z, x 2024-03-26T04:09:31.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:31 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:09:31.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:31 smithi149 ceph-mon[30657]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:31.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:31 smithi204 ceph-mon[30652]: mgrmap e15: y(active, since 83s), standbys: z, x 2024-03-26T04:09:31.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:31 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:09:31.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:31 smithi204 ceph-mon[30652]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:33.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:33 smithi204 ceph-mon[30652]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:33.758 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:33 smithi136 ceph-mon[25848]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:33.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:33 smithi149 ceph-mon[30657]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:35.768 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:35 smithi136 ceph-mon[25848]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:35.768 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:09:35.768 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:35.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:35 smithi149 ceph-mon[30657]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:35.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:35 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:09:35.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:35 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:35.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:35 smithi204 ceph-mon[30652]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:35.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:35 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:09:35.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:35 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:36.694 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:36 smithi136 ceph-mon[25848]: Deploying daemon osd.0 on smithi136 2024-03-26T04:09:36.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:36 smithi149 ceph-mon[30657]: Deploying daemon osd.0 on smithi136 2024-03-26T04:09:36.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:36 smithi204 ceph-mon[30652]: Deploying daemon osd.0 on smithi136 2024-03-26T04:09:37.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:37 smithi136 ceph-mon[25848]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:37.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:37 smithi149 ceph-mon[30657]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:37.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:37 smithi204 ceph-mon[30652]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:39.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:39 smithi136 ceph-mon[25848]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:39.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:39 smithi149 ceph-mon[30657]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:39.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:39 smithi204 ceph-mon[30652]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:41.555 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:41 smithi136 ceph-mon[25848]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:41.556 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:41.556 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:41.556 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:41.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:41 smithi149 ceph-mon[30657]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:41.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:41.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:41.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:41.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:41 smithi204 ceph-mon[30652]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:41.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:41.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:41.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:43.836 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:43 smithi136 ceph-mon[25848]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:43.836 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:43 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:43.836 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:43 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:43.836 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:43 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:43.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:43 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:43.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:43 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:43.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:43 smithi149 ceph-mon[30657]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:43.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:43 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:43.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:43 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:43.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:43 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:43.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:43 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:43.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:43 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:43.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:43 smithi204 ceph-mon[30652]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:43.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:43 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:43.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:43 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:43.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:43 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:43.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:43 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:43.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:43 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:44.494 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 0 on host 'smithi136' 2024-03-26T04:09:44.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:44 smithi136 ceph-mon[25848]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:09:44.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:44 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:44.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:44 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:44.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:44 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:44.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:44 smithi149 ceph-mon[30657]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:09:44.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:44 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:44.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:44 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:44.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:44 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:44.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:44 smithi204 ceph-mon[30652]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:09:44.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:44 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:44.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:44 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:44.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:44 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:45.142 DEBUG:teuthology.orchestra.run.smithi136:osd.0> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T04:09:45.145 INFO:tasks.cephadm:Deploying osd.1 on smithi149 with /dev/nvme4n1... 2024-03-26T04:09:45.145 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T04:09:45.390 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:09:45.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:45 smithi136 ceph-mon[25848]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:45.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:45 smithi136 ceph-mon[25848]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:09:45.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:45 smithi136 ceph-mon[25848]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:09:45.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:45 smithi136 ceph-mon[25848]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T04:09:45.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:45 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:45.820 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:45 smithi149 ceph-mon[30657]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:45.820 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:45 smithi149 ceph-mon[30657]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:09:45.820 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:45 smithi149 ceph-mon[30657]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:09:45.820 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:45 smithi149 ceph-mon[30657]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T04:09:45.820 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:45 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:45.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:45 smithi204 ceph-mon[30652]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:45.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:45 smithi204 ceph-mon[30652]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:09:45.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:45 smithi204 ceph-mon[30652]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:09:45.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:45 smithi204 ceph-mon[30652]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T04:09:45.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:45 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:46.068 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:09:45 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0[44214]: 2024-03-26T04:09:45.825+0000 7f3f4fe45700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T04:09:46.068 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:09:45 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0[44214]: 2024-03-26T04:09:45.828+0000 7f3f47c30700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:09:46.636 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:46 smithi149 ceph-mon[30657]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T04:09:46.636 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:46 smithi149 ceph-mon[30657]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:09:46.636 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:46 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:46.636 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:46 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:46.636 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:46 smithi149 ceph-mon[30657]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' 2024-03-26T04:09:46.648 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:46 smithi136 ceph-mon[25848]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T04:09:46.648 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:46 smithi136 ceph-mon[25848]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:09:46.648 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:46 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:46.648 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:46 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:46.648 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:46 smithi136 ceph-mon[25848]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' 2024-03-26T04:09:46.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:46 smithi204 ceph-mon[30652]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T04:09:46.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:46 smithi204 ceph-mon[30652]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:09:46.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:46 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:46.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:46 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:46.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:46 smithi204 ceph-mon[30652]: from='osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744]' entity='osd.0' 2024-03-26T04:09:47.174 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T04:09:47.175 INFO:teuthology.orchestra.run.smithi149.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T04:09:47.175 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T04:09:47.175 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-03-26T04:09:47.175 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-03-26T04:09:47.175 INFO:teuthology.orchestra.run.smithi149.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0189889 s, 552 MB/s 2024-03-26T04:09:47.175 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-03-26T04:09:47.737 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:47 smithi149 ceph-mon[30657]: purged_snaps scrub starts 2024-03-26T04:09:47.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:47 smithi149 ceph-mon[30657]: purged_snaps scrub ok 2024-03-26T04:09:47.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:47 smithi149 ceph-mon[30657]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:47.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:47.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:47 smithi149 ceph-mon[30657]: osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744] boot 2024-03-26T04:09:47.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:47 smithi149 ceph-mon[30657]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:09:47.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:47.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:47 smithi204 ceph-mon[30652]: purged_snaps scrub starts 2024-03-26T04:09:47.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:47 smithi204 ceph-mon[30652]: purged_snaps scrub ok 2024-03-26T04:09:47.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:47 smithi204 ceph-mon[30652]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:47.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:47.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:47 smithi204 ceph-mon[30652]: osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744] boot 2024-03-26T04:09:47.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:47 smithi204 ceph-mon[30652]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:09:47.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:47.793 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi149:/dev/nvme4n1 2024-03-26T04:09:47.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:47 smithi136 ceph-mon[25848]: purged_snaps scrub starts 2024-03-26T04:09:47.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:47 smithi136 ceph-mon[25848]: purged_snaps scrub ok 2024-03-26T04:09:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:47 smithi136 ceph-mon[25848]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:09:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:47 smithi136 ceph-mon[25848]: osd.0 [v2:172.21.15.136:6802/4198578744,v1:172.21.15.136:6803/4198578744] boot 2024-03-26T04:09:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:47 smithi136 ceph-mon[25848]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:09:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:09:48.026 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:09:49.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: Detected new or changed devices on smithi136 2024-03-26T04:09:49.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:09:49.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: Adjusting osd_memory_target on smithi136 to 17088M 2024-03-26T04:09:49.480 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.481 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:49.481 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:49.481 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.481 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:49.481 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:49 smithi149 ceph-mon[30657]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:09:49.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: Detected new or changed devices on smithi136 2024-03-26T04:09:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:09:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: Adjusting osd_memory_target on smithi136 to 17088M 2024-03-26T04:09:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:49 smithi136 ceph-mon[25848]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:09:49.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: Detected new or changed devices on smithi136 2024-03-26T04:09:49.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:09:49.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: Adjusting osd_memory_target on smithi136 to 17088M 2024-03-26T04:09:49.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:49.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:09:49.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:09:49.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:49.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:49 smithi204 ceph-mon[30652]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:09:50.632 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:50 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:09:50.633 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:50 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:09:50.633 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:50 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:50.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:50 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:09:50.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:50 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:09:50.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:50 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:50.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:50 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:09:50.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:50 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:09:50.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:50 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:51.575 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:51 smithi149 ceph-mon[30657]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:51.575 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:51 smithi149 ceph-mon[30657]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:51.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:51 smithi136 ceph-mon[25848]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:51.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:51 smithi136 ceph-mon[25848]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:51.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:51 smithi204 ceph-mon[30652]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:09:51.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:51 smithi204 ceph-mon[30652]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:52.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:52 smithi149 ceph-mon[30657]: from='client.? 172.21.15.149:0/1625372406' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00b6ef7-9fd4-402b-a04d-5db2fea2d01c"}]: dispatch 2024-03-26T04:09:52.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:52 smithi149 ceph-mon[30657]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00b6ef7-9fd4-402b-a04d-5db2fea2d01c"}]: dispatch 2024-03-26T04:09:52.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:52 smithi149 ceph-mon[30657]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b00b6ef7-9fd4-402b-a04d-5db2fea2d01c"}]': finished 2024-03-26T04:09:52.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:52 smithi149 ceph-mon[30657]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:09:52.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:52 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:09:52.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:52 smithi204 ceph-mon[30652]: from='client.? 172.21.15.149:0/1625372406' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00b6ef7-9fd4-402b-a04d-5db2fea2d01c"}]: dispatch 2024-03-26T04:09:52.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:52 smithi204 ceph-mon[30652]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00b6ef7-9fd4-402b-a04d-5db2fea2d01c"}]: dispatch 2024-03-26T04:09:52.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:52 smithi204 ceph-mon[30652]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b00b6ef7-9fd4-402b-a04d-5db2fea2d01c"}]': finished 2024-03-26T04:09:52.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:52 smithi204 ceph-mon[30652]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:09:52.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:52 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:09:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:52 smithi136 ceph-mon[25848]: from='client.? 172.21.15.149:0/1625372406' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00b6ef7-9fd4-402b-a04d-5db2fea2d01c"}]: dispatch 2024-03-26T04:09:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:52 smithi136 ceph-mon[25848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00b6ef7-9fd4-402b-a04d-5db2fea2d01c"}]: dispatch 2024-03-26T04:09:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:52 smithi136 ceph-mon[25848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b00b6ef7-9fd4-402b-a04d-5db2fea2d01c"}]': finished 2024-03-26T04:09:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:52 smithi136 ceph-mon[25848]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:09:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:52 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:09:53.737 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:53 smithi149 ceph-mon[30657]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:53.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:53 smithi204 ceph-mon[30652]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:53.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:53 smithi136 ceph-mon[25848]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:54.737 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:54 smithi149 ceph-mon[30657]: from='client.? 172.21.15.149:0/713008537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:09:54.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:54 smithi204 ceph-mon[30652]: from='client.? 172.21.15.149:0/713008537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:09:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:54 smithi136 ceph-mon[25848]: from='client.? 172.21.15.149:0/713008537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:09:55.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:55 smithi149 ceph-mon[30657]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:55.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:55 smithi204 ceph-mon[30652]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:55.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:55 smithi136 ceph-mon[25848]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:57.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:57 smithi149 ceph-mon[30657]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:57.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:57 smithi204 ceph-mon[30652]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:57 smithi136 ceph-mon[25848]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:59.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:59 smithi204 ceph-mon[30652]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:59.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:59 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:09:59.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:09:59 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:59.937 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:59 smithi149 ceph-mon[30657]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:59.937 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:59 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:09:59.937 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:09:59 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:09:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:59 smithi136 ceph-mon[25848]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:09:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:59 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:09:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:09:59 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:00.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:00 smithi204 ceph-mon[30652]: Deploying daemon osd.1 on smithi149 2024-03-26T04:10:00.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:00 smithi204 ceph-mon[30652]: overall HEALTH_OK 2024-03-26T04:10:00.850 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:00 smithi149 ceph-mon[30657]: Deploying daemon osd.1 on smithi149 2024-03-26T04:10:00.850 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:00 smithi149 ceph-mon[30657]: overall HEALTH_OK 2024-03-26T04:10:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:00 smithi136 ceph-mon[25848]: Deploying daemon osd.1 on smithi149 2024-03-26T04:10:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:00 smithi136 ceph-mon[25848]: overall HEALTH_OK 2024-03-26T04:10:01.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:01 smithi149 ceph-mon[30657]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:01.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:01 smithi204 ceph-mon[30652]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:01.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:01 smithi136 ceph-mon[25848]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:03.881 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:03 smithi149 ceph-mon[30657]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:03 smithi136 ceph-mon[25848]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:03.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:03 smithi204 ceph-mon[30652]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:05.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:05 smithi149 ceph-mon[30657]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:05.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:05 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:05.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:05 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:05.604 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:05 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:05 smithi136 ceph-mon[25848]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:05 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:05 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:05 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:05.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:05 smithi204 ceph-mon[30652]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:05.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:05 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:05.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:05 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:05.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:05 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:07.789 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:07 smithi149 ceph-mon[30657]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:07.789 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:07 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:07.789 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:07 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:07.789 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:07 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:07.790 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:07 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:07.790 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:07 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:07.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:07 smithi136 ceph-mon[25848]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:07.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:07.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:07.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:07.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:07.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:07 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:07.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:07 smithi204 ceph-mon[30652]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:07.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:07.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:07.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:07.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:07.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:07 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:08.300 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 1 on host 'smithi149' 2024-03-26T04:10:08.854 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:08 smithi149 ceph-mon[30657]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:10:08.854 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:08 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:08.854 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:08 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:08.854 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:08 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:08.879 DEBUG:teuthology.orchestra.run.smithi149:osd.1> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T04:10:08.882 INFO:tasks.cephadm:Deploying osd.2 on smithi204 with /dev/nvme4n1... 2024-03-26T04:10:08.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T04:10:08.912 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:08 smithi204 ceph-mon[30652]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:10:08.912 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:08 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:08.912 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:08 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:08.912 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:08 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:08.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:08 smithi136 ceph-mon[25848]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:10:08.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:08 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:08.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:08 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:08.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:08 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:09.125 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:10:09.849 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:09 smithi136 ceph-mon[25848]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:09.850 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:09 smithi136 ceph-mon[25848]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:10:09.850 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:09 smithi136 ceph-mon[25848]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:10:09.850 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:09 smithi136 ceph-mon[25848]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-26T04:10:09.850 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:09 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:09.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:09 smithi204 ceph-mon[30652]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:09.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:09 smithi204 ceph-mon[30652]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:10:09.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:09 smithi204 ceph-mon[30652]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:10:09.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:09 smithi204 ceph-mon[30652]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-26T04:10:09.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:09 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:09.893 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:09 smithi149 ceph-mon[30657]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:09.893 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:09 smithi149 ceph-mon[30657]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:10:09.893 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:09 smithi149 ceph-mon[30657]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:10:09.893 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:09 smithi149 ceph-mon[30657]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-03-26T04:10:09.893 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:09 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:09.893 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:10:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1[35231]: 2024-03-26T04:10:09.527+0000 7efdf8185700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T04:10:09.893 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:10:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1[35231]: 2024-03-26T04:10:09.530+0000 7efdf3f78700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:10:10.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:10 smithi204 ceph-mon[30652]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-26T04:10:10.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:10 smithi204 ceph-mon[30652]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:10:10.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:10.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:10.937 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T04:10:10.937 INFO:teuthology.orchestra.run.smithi204.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T04:10:10.937 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T04:10:10.937 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-03-26T04:10:10.937 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-03-26T04:10:10.937 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140016 s, 749 MB/s 2024-03-26T04:10:10.937 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-03-26T04:10:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:10 smithi136 ceph-mon[25848]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-26T04:10:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:10 smithi136 ceph-mon[25848]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:10:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:10.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:10 smithi149 ceph-mon[30657]: from='osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-03-26T04:10:10.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:10 smithi149 ceph-mon[30657]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:10:10.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:10.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:11.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi204:/dev/nvme4n1 2024-03-26T04:10:11.840 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:10:11.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:11 smithi136 ceph-mon[25848]: purged_snaps scrub starts 2024-03-26T04:10:11.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:11 smithi136 ceph-mon[25848]: purged_snaps scrub ok 2024-03-26T04:10:11.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:11 smithi136 ceph-mon[25848]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:11.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:11 smithi136 ceph-mon[25848]: osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559] boot 2024-03-26T04:10:11.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:11 smithi136 ceph-mon[25848]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:10:11.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:11 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:11.961 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:11 smithi204 ceph-mon[30652]: purged_snaps scrub starts 2024-03-26T04:10:11.961 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:11 smithi204 ceph-mon[30652]: purged_snaps scrub ok 2024-03-26T04:10:11.961 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:11 smithi204 ceph-mon[30652]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:11.962 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:11 smithi204 ceph-mon[30652]: osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559] boot 2024-03-26T04:10:11.962 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:11 smithi204 ceph-mon[30652]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:10:11.962 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:11 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:11.966 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:11 smithi149 ceph-mon[30657]: purged_snaps scrub starts 2024-03-26T04:10:11.966 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:11 smithi149 ceph-mon[30657]: purged_snaps scrub ok 2024-03-26T04:10:11.966 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:11 smithi149 ceph-mon[30657]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:10:11.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:11 smithi149 ceph-mon[30657]: osd.1 [v2:172.21.15.149:6800/883566559,v1:172.21.15.149:6801/883566559] boot 2024-03-26T04:10:11.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:11 smithi149 ceph-mon[30657]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:10:11.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:11 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:10:12.935 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:12 smithi204 ceph-mon[30652]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:10:12.935 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:10:12.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:12.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:12.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:12 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:12 smithi136 ceph-mon[25848]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:10:12.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:10:12.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:12.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:12.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:12 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:12 smithi149 ceph-mon[30657]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:10:12.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:10:12.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:12.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:12.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:12.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:12 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:13 smithi136 ceph-mon[25848]: Detected new or changed devices on smithi149 2024-03-26T04:10:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:13 smithi136 ceph-mon[25848]: Adjusting osd_memory_target on smithi149 to 5816M 2024-03-26T04:10:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:13 smithi136 ceph-mon[25848]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:13.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:13 smithi149 ceph-mon[30657]: Detected new or changed devices on smithi149 2024-03-26T04:10:13.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:13 smithi149 ceph-mon[30657]: Adjusting osd_memory_target on smithi149 to 5816M 2024-03-26T04:10:13.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:13 smithi149 ceph-mon[30657]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:13.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:13 smithi204 ceph-mon[30652]: Detected new or changed devices on smithi149 2024-03-26T04:10:13.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:13 smithi204 ceph-mon[30652]: Adjusting osd_memory_target on smithi149 to 5816M 2024-03-26T04:10:13.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:13 smithi204 ceph-mon[30652]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:14.861 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:14 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:10:14.861 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:14 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:10:14.861 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:14 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:14.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:14 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:10:14.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:14 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:10:14.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:14 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:14 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:10:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:14 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:10:14.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:14 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:15.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:15 smithi136 ceph-mon[25848]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:15.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:15 smithi136 ceph-mon[25848]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:15.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:15 smithi149 ceph-mon[30657]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:15.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:15 smithi149 ceph-mon[30657]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:15.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:15 smithi204 ceph-mon[30652]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:15.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:15 smithi204 ceph-mon[30652]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:16.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:16 smithi204 ceph-mon[30652]: from='client.? 172.21.15.204:0/401583404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faeabeb6-c22b-4832-ae19-1e44f9524e23"}]: dispatch 2024-03-26T04:10:16.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:16 smithi204 ceph-mon[30652]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faeabeb6-c22b-4832-ae19-1e44f9524e23"}]: dispatch 2024-03-26T04:10:16.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:16 smithi204 ceph-mon[30652]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "faeabeb6-c22b-4832-ae19-1e44f9524e23"}]': finished 2024-03-26T04:10:16.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:16 smithi204 ceph-mon[30652]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:10:16.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:16 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:16.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:16 smithi136 ceph-mon[25848]: from='client.? 172.21.15.204:0/401583404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faeabeb6-c22b-4832-ae19-1e44f9524e23"}]: dispatch 2024-03-26T04:10:16.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:16 smithi136 ceph-mon[25848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faeabeb6-c22b-4832-ae19-1e44f9524e23"}]: dispatch 2024-03-26T04:10:16.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:16 smithi136 ceph-mon[25848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "faeabeb6-c22b-4832-ae19-1e44f9524e23"}]': finished 2024-03-26T04:10:16.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:16 smithi136 ceph-mon[25848]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:10:16.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:16 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:16.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:16 smithi149 ceph-mon[30657]: from='client.? 172.21.15.204:0/401583404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faeabeb6-c22b-4832-ae19-1e44f9524e23"}]: dispatch 2024-03-26T04:10:16.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:16 smithi149 ceph-mon[30657]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faeabeb6-c22b-4832-ae19-1e44f9524e23"}]: dispatch 2024-03-26T04:10:16.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:16 smithi149 ceph-mon[30657]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "faeabeb6-c22b-4832-ae19-1e44f9524e23"}]': finished 2024-03-26T04:10:16.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:16 smithi149 ceph-mon[30657]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:10:16.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:16 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:17.910 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:17 smithi204 ceph-mon[30652]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:17.911 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:17 smithi204 ceph-mon[30652]: from='client.? 172.21.15.204:0/365543002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:10:17.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:17 smithi136 ceph-mon[25848]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:17.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:17 smithi136 ceph-mon[25848]: from='client.? 172.21.15.204:0/365543002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:10:17.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:17 smithi149 ceph-mon[30657]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:17.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:17 smithi149 ceph-mon[30657]: from='client.? 172.21.15.204:0/365543002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:10:19.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:19 smithi136 ceph-mon[25848]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:19.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:19 smithi149 ceph-mon[30657]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:19.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:19 smithi204 ceph-mon[30652]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:21.938 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:21 smithi149 ceph-mon[30657]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:21.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:21 smithi136 ceph-mon[25848]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:21.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:21 smithi204 ceph-mon[30652]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:23.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:23 smithi204 ceph-mon[30652]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:23.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:23 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:10:23.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:23 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:23 smithi136 ceph-mon[25848]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:10:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:23 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:23.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:23 smithi149 ceph-mon[30657]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:23.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:23 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:10:23.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:23 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:24.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:24 smithi204 ceph-mon[30652]: Deploying daemon osd.2 on smithi204 2024-03-26T04:10:24.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:24 smithi136 ceph-mon[25848]: Deploying daemon osd.2 on smithi204 2024-03-26T04:10:24.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:24 smithi149 ceph-mon[30657]: Deploying daemon osd.2 on smithi204 2024-03-26T04:10:25.821 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:25 smithi204 ceph-mon[30652]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:25.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:25 smithi136 ceph-mon[25848]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:25.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:25 smithi149 ceph-mon[30657]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:27.879 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:27 smithi204 ceph-mon[30652]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:27.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:27 smithi136 ceph-mon[25848]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:27.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:27 smithi149 ceph-mon[30657]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:29.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:29 smithi204 ceph-mon[30652]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:29.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:29 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:29.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:29 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:29.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:29 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:29.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:29 smithi136 ceph-mon[25848]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:29.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:29 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:29.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:29 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:29.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:29 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:29.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:29 smithi149 ceph-mon[30657]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:29.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:29 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:29.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:29 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:29.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:29 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:31.737 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:31 smithi149 ceph-mon[30657]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:31.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:31 smithi204 ceph-mon[30652]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:31 smithi136 ceph-mon[25848]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:32.146 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 2 on host 'smithi204' 2024-03-26T04:10:32.727 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:32 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.728 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:32 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.728 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:32 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:32.728 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:32 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:32.728 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:32 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.728 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:32 smithi204 ceph-mon[30652]: from='osd.2 [v2:172.21.15.204:6800/3457499621,v1:172.21.15.204:6801/3457499621]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:10:32.728 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:32 smithi204 ceph-mon[30652]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:10:32.728 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:32 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.728 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:32 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.728 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:32 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:32.753 DEBUG:teuthology.orchestra.run.smithi204:osd.2> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T04:10:32.755 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-26T04:10:32.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T04:10:32.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:32 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:32 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:32 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:32.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:32 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:32.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:32 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:32 smithi136 ceph-mon[25848]: from='osd.2 [v2:172.21.15.204:6800/3457499621,v1:172.21.15.204:6801/3457499621]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:10:32.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:32 smithi136 ceph-mon[25848]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:10:32.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:32 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:32 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:32 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:32.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:32 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:32 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:32 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:32.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:32 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:32.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:32 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:32 smithi149 ceph-mon[30657]: from='osd.2 [v2:172.21.15.204:6800/3457499621,v1:172.21.15.204:6801/3457499621]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:10:32.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:32 smithi149 ceph-mon[30657]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:10:32.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:32 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:32 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:32.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:32 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:33.001 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:33.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:33 smithi136 ceph-mon[25848]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:33.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:33 smithi136 ceph-mon[25848]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:10:33.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:33 smithi136 ceph-mon[25848]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:10:33.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:33 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:33.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:33 smithi136 ceph-mon[25848]: from='osd.2 [v2:172.21.15.204:6800/3457499621,v1:172.21.15.204:6801/3457499621]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-26T04:10:33.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:33 smithi136 ceph-mon[25848]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-26T04:10:33.985 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:33 smithi204 ceph-mon[30652]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:33.985 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:33 smithi204 ceph-mon[30652]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:10:33.985 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:33 smithi204 ceph-mon[30652]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:10:33.985 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:33 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:33.985 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:33 smithi204 ceph-mon[30652]: from='osd.2 [v2:172.21.15.204:6800/3457499621,v1:172.21.15.204:6801/3457499621]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-26T04:10:33.985 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:33 smithi204 ceph-mon[30652]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-26T04:10:33.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:33 smithi149 ceph-mon[30657]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:33.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:33 smithi149 ceph-mon[30657]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:10:33.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:33 smithi149 ceph-mon[30657]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:10:33.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:33 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:33.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:33 smithi149 ceph-mon[30657]: from='osd.2 [v2:172.21.15.204:6800/3457499621,v1:172.21.15.204:6801/3457499621]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-26T04:10:33.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:33 smithi149 ceph-mon[30657]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-26T04:10:34.247 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:10:34 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2[35910]: 2024-03-26T04:10:34.045+0000 7fbfb3997700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T04:10:34.248 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:10:34 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2[35910]: 2024-03-26T04:10:34.048+0000 7fbfaaf81700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:10:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:34 smithi136 ceph-mon[25848]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-26T04:10:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:34 smithi136 ceph-mon[25848]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:10:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:34 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:34 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:34 smithi136 ceph-mon[25848]: from='osd.2 ' entity='osd.2' 2024-03-26T04:10:34.843 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:10:34.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:34 smithi149 ceph-mon[30657]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-26T04:10:34.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:34 smithi149 ceph-mon[30657]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:10:34.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:34 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:34.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:34 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:34.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:34 smithi149 ceph-mon[30657]: from='osd.2 ' entity='osd.2' 2024-03-26T04:10:34.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:34 smithi204 ceph-mon[30652]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-26T04:10:34.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:34 smithi204 ceph-mon[30652]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:10:34.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:34 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:34.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:34 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:34.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:34 smithi204 ceph-mon[30652]: from='osd.2 ' entity='osd.2' 2024-03-26T04:10:35.457 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":17,"num_osds":3,"num_up_osds":2,"osd_up_since":1711426210,"num_in_osds":3,"osd_in_since":1711426216,"num_remapped_pgs":0} 2024-03-26T04:10:35.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:35 smithi136 ceph-mon[25848]: purged_snaps scrub starts 2024-03-26T04:10:35.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:35 smithi136 ceph-mon[25848]: purged_snaps scrub ok 2024-03-26T04:10:35.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:35 smithi136 ceph-mon[25848]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:35.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:35.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:35 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/3467375025' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:10:35.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:35 smithi136 ceph-mon[25848]: osd.2 [v2:172.21.15.204:6800/3457499621,v1:172.21.15.204:6801/3457499621] boot 2024-03-26T04:10:35.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:35 smithi136 ceph-mon[25848]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:10:35.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:35 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:35.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:35 smithi204 ceph-mon[30652]: purged_snaps scrub starts 2024-03-26T04:10:35.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:35 smithi204 ceph-mon[30652]: purged_snaps scrub ok 2024-03-26T04:10:35.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:35 smithi204 ceph-mon[30652]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:35.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:35 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:35.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:35 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/3467375025' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:10:35.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:35 smithi204 ceph-mon[30652]: osd.2 [v2:172.21.15.204:6800/3457499621,v1:172.21.15.204:6801/3457499621] boot 2024-03-26T04:10:35.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:35 smithi204 ceph-mon[30652]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:10:35.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:35 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:35.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:35 smithi149 ceph-mon[30657]: purged_snaps scrub starts 2024-03-26T04:10:35.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:35 smithi149 ceph-mon[30657]: purged_snaps scrub ok 2024-03-26T04:10:35.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:35 smithi149 ceph-mon[30657]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:10:35.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:35 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:35.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:35 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/3467375025' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:10:35.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:35 smithi149 ceph-mon[30657]: osd.2 [v2:172.21.15.204:6800/3457499621,v1:172.21.15.204:6801/3457499621] boot 2024-03-26T04:10:35.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:35 smithi149 ceph-mon[30657]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:10:35.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:35 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:10:36.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T04:10:36.710 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:37.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: Detected new or changed devices on smithi204 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: Adjusting osd_memory_target on smithi204 to 17088M 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:10:37.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:10:37.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:10:37.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:36 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: Detected new or changed devices on smithi204 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: Adjusting osd_memory_target on smithi204 to 17088M 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:10:37.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:10:37.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:10:37.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:36 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:10:37.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: Detected new or changed devices on smithi204 2024-03-26T04:10:37.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:10:37.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: Adjusting osd_memory_target on smithi204 to 17088M 2024-03-26T04:10:37.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:37.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:37.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:37.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:37.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:10:37.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:10:37.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:10:37.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:36 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:10:38.385 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:10:38 smithi136 sudo[49682]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T04:10:38.385 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:10:38 smithi136 sudo[49682]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:38.385 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:10:38 smithi136 sudo[49682]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:38.385 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:10:38 smithi136 sudo[49682]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:38.609 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:10:38.698 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:10:38 smithi136 sudo[49710]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T04:10:38.698 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:10:38 smithi136 sudo[49710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:38.699 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:10:38 smithi136 sudo[49710]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:38.699 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:10:38 smithi136 sudo[49710]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:38.915 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:10:38 smithi149 sudo[40165]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T04:10:38.916 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:10:38 smithi149 sudo[40165]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:38.916 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:10:38 smithi149 sudo[40165]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:38.916 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:10:38 smithi149 sudo[40165]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:38.916 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:38 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:10:38.916 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:38 smithi149 ceph-mon[30657]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:10:38.916 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:38 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:38.916 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:38 smithi149 ceph-mon[30657]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:38.916 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:38 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/146118365' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:10:39.167 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:38 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:10:39.167 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:38 smithi136 ceph-mon[25848]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:10:39.167 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:38 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:39.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:38 smithi136 ceph-mon[25848]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:39.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:38 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/146118365' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:10:39.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:38 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:10:39.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:38 smithi204 ceph-mon[30652]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:10:39.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:38 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:39.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:38 smithi204 ceph-mon[30652]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:39.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:38 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/146118365' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:10:39.193 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":20,"num_osds":3,"num_up_osds":3,"osd_up_since":1711426235,"num_in_osds":3,"osd_in_since":1711426216,"num_remapped_pgs":0} 2024-03-26T04:10:39.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:10:39.238 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:10:38 smithi149 sudo[40168]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T04:10:39.238 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:10:38 smithi149 sudo[40168]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:39.238 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:10:38 smithi149 sudo[40168]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:39.238 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:10:38 smithi149 sudo[40168]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:39.429 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:39.434 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:10:39 smithi204 sudo[41212]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T04:10:39.434 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:10:39 smithi204 sudo[41212]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:39.434 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:10:39 smithi204 sudo[41212]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:39.434 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:10:39 smithi204 sudo[41212]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:39.747 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:10:39 smithi204 sudo[41215]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T04:10:39.747 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:10:39 smithi204 sudo[41215]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:39.747 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:10:39 smithi204 sudo[41215]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:39.748 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:10:39 smithi204 sudo[41215]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:39.812 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 sudo[49850]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:10:39.812 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 sudo[49850]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:39.812 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 sudo[49850]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:39.812 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 sudo[49850]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:40.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 sudo[49870]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:10:40.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 sudo[49870]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:40.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 sudo[49870]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:40.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 sudo[49870]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:40.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 ceph-mon[25848]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:10:40.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:40.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:40.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:39 smithi136 ceph-mon[25848]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:10:40.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:39 smithi149 ceph-mon[30657]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:10:40.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:39 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:40.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:39 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:40.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:39 smithi149 ceph-mon[30657]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:10:40.242 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:39 smithi204 ceph-mon[30652]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:10:40.242 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:39 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:40.242 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:39 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:40.242 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:39 smithi204 ceph-mon[30652]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:10:40.496 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:40 smithi204 sudo[41218]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:10:40.496 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:40 smithi204 sudo[41218]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:40.496 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:40 smithi204 sudo[41218]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:40.496 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:40 smithi204 sudo[41218]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:40.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:40 smithi204 sudo[41221]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:10:40.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:40 smithi204 sudo[41221]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:40.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:40 smithi204 sudo[41221]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:40.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:40 smithi204 sudo[41221]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:41.012 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:40 smithi149 sudo[40171]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:10:41.012 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:40 smithi149 sudo[40171]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:41.012 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:40 smithi149 sudo[40171]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:41.012 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:40 smithi149 sudo[40171]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:41.099 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:10:41.099 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:10:41.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:41 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:10:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:10:41.255 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":21,"fsid":"276a1e6c-eb26-11ee-95d0-87774f69a715","created":"2024-03-26T04:07:10.618462+0000","modified":"2024-03-26T04:10:38.780850+0000","last_up_change":"2024-03-26T04:10:35.047970+0000","last_in_change":"2024-03-26T04:10:16.269354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:10:36.777571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8bc9716b-db6c-4d3a-a77b-0424e38088a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6803","nonce":4198578744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6805","nonce":4198578744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6809","nonce":4198578744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6807","nonce":4198578744}]},"public_addr":"172.21.15.136:6803/4198578744","cluster_addr":"172.21.15.136:6805/4198578744","heartbeat_back_addr":"172.21.15.136:6809/4198578744","heartbeat_front_addr":"172.21.15.136:6807/4198578744","state":["exists","up"]},{"osd":1,"uuid":"b00b6ef7-9fd4-402b-a04d-5db2fea2d01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6801","nonce":883566559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6803","nonce":883566559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6807","nonce":883566559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6805","nonce":883566559}]},"public_addr":"172.21.15.149:6801/883566559","cluster_addr":"172.21.15.149:6803/883566559","heartbeat_back_addr":"172.21.15.149:6807/883566559","heartbeat_front_addr":"172.21.15.149:6805/883566559","state":["exists","up"]},{"osd":2,"uuid":"faeabeb6-c22b-4832-ae19-1e44f9524e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6801","nonce":3457499621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6803","nonce":3457499621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6807","nonce":3457499621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6805","nonce":3457499621}]},"public_addr":"172.21.15.204:6801/3457499621","cluster_addr":"172.21.15.204:6803/3457499621","heartbeat_back_addr":"172.21.15.204:6807/3457499621","heartbeat_front_addr":"172.21.15.204:6805/3457499621","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:09:44.740827+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:10:08.766950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:10:32.825143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:6801/3121969854":"2024-03-27T04:08:06.425886+0000","172.21.15.136:6800/3121969854":"2024-03-27T04:08:06.425886+0000","172.21.15.136:0/2374692480":"2024-03-27T04:08:06.425886+0000","172.21.15.136:0/2817395307":"2024-03-27T04:07:39.177832+0000","172.21.15.136:0/3540036637":"2024-03-27T04:08:06.425886+0000","172.21.15.136:6801/391737807":"2024-03-27T04:07:39.177832+0000","172.21.15.136:0/3458371792":"2024-03-27T04:07:39.177832+0000","172.21.15.136:0/4153123335":"2024-03-27T04:08:06.425886+0000","172.21.15.136:0/1442132325":"2024-03-27T04:07:39.177832+0000","172.21.15.136:6800/391737807":"2024-03-27T04:07:39.177832+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:10:41.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 sudo[40174]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:10:41.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 sudo[40174]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:10:41.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 sudo[40174]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:10:41.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 sudo[40174]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:10:41.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:10:41.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:10:41.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:10:41.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:10:41.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:10:41.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:41 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:10:41.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:10:41.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:10:41.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:10:41.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:41 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:10:41.881 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:10:36.777571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T04:10:41.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T04:10:42.121 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:42.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:42 smithi136 ceph-mon[25848]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:10:42.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:42 smithi136 ceph-mon[25848]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:10:42.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:42 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:42.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:42 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/2064501384' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:10:42.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:42 smithi149 ceph-mon[30657]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:10:42.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:42 smithi149 ceph-mon[30657]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:10:42.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:42 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:42.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:42 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/2064501384' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:10:42.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:42 smithi204 ceph-mon[30652]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:10:42.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:42 smithi204 ceph-mon[30652]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:10:42.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:42 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:42.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:42 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/2064501384' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:10:43.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:43 smithi136 ceph-mon[25848]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:43.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:43 smithi149 ceph-mon[30657]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:43.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:43 smithi204 ceph-mon[30652]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:43.930 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-03-26T04:10:44.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:44 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/2319062964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:10:44.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:44 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/2319062964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:10:44.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:44 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/2319062964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:10:44.520 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi136 2024-03-26T04:10:44.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi136=foo.a' 2024-03-26T04:10:44.767 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:10:45.210 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:45 smithi204 ceph-mon[30652]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:45.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:45 smithi136 ceph-mon[25848]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:45.487 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:45 smithi149 ceph-mon[30657]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:10:46.513 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled rgw.foo.a update... 2024-03-26T04:10:47.162 DEBUG:teuthology.orchestra.run.smithi136:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T04:10:47.165 INFO:tasks.cephadm:Adding prometheus.a on smithi149 2024-03-26T04:10:47.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi149=a' 2024-03-26T04:10:47.418 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:10:47.696 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi136=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:47.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:47 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:47.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-26T04:10:47.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi136=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:47.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T04:10:47.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:10:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:10:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:47.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:47 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:47.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi136=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:47.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:47 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:48.820 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:48 smithi204 ceph-mon[30652]: Deploying daemon rgw.foo.a on smithi136 2024-03-26T04:10:48.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:48 smithi136 ceph-mon[25848]: Deploying daemon rgw.foo.a on smithi136 2024-03-26T04:10:48.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:48 smithi149 ceph-mon[30657]: Deploying daemon rgw.foo.a on smithi136 2024-03-26T04:10:49.146 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled prometheus update... 2024-03-26T04:10:49.520 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:10:49 smithi136 systemd[1]: Starting Ceph rgw.foo.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:10:49.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:49 smithi204 ceph-mon[30652]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-26T04:10:49.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:49 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:49.780 DEBUG:teuthology.orchestra.run.smithi149:prometheus.a> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T04:10:49.783 INFO:tasks.cephadm:Adding node-exporter.a on smithi136 2024-03-26T04:10:49.783 INFO:tasks.cephadm:Adding node-exporter.b on smithi149 2024-03-26T04:10:49.783 INFO:tasks.cephadm:Adding node-exporter.c on smithi204 2024-03-26T04:10:49.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '3;smithi136=a;smithi149=b;smithi204=c' 2024-03-26T04:10:49.810 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:49 smithi149 ceph-mon[30657]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-26T04:10:49.811 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:49 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:49.854 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:49 smithi136 ceph-mon[25848]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-26T04:10:49.854 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:49 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:49.854 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:10:49 smithi136 podman[50502]: 2024-03-26 04:10:49.546214068 +0000 UTC m=+0.017588648 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:10:49.854 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:10:49 smithi136 podman[50502]: 2024-03-26 04:10:49.643662934 +0000 UTC m=+0.115037515 container create d4d28675bacd9d50ca1e81ffb940ea293bdf2eca1d2d7beb89f60bb283bde269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-26T04:10:49.854 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:10:49 smithi136 podman[50502]: 2024-03-26 04:10:49.854415051 +0000 UTC m=+0.325789631 container init d4d28675bacd9d50ca1e81ffb940ea293bdf2eca1d2d7beb89f60bb283bde269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-rgw-foo-a, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T04:10:50.017 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:10:50.198 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:10:49 smithi136 podman[50502]: 2024-03-26 04:10:49.860091032 +0000 UTC m=+0.331465613 container start d4d28675bacd9d50ca1e81ffb940ea293bdf2eca1d2d7beb89f60bb283bde269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True) 2024-03-26T04:10:50.198 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:10:49 smithi136 bash[50502]: d4d28675bacd9d50ca1e81ffb940ea293bdf2eca1d2d7beb89f60bb283bde269 2024-03-26T04:10:50.198 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:10:49 smithi136 systemd[1]: Started Ceph rgw.foo.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:10:50.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:50 smithi204 ceph-mon[30652]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:50.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:50 smithi204 ceph-mon[30652]: Saving service prometheus spec with placement smithi149=a;count:1 2024-03-26T04:10:50.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:50 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:50 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:50 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:50 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:50 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:50 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:50.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:50 smithi136 ceph-mon[25848]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:50.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:50 smithi136 ceph-mon[25848]: Saving service prometheus spec with placement smithi149=a;count:1 2024-03-26T04:10:50.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:50 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:50 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:50 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:50 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:50 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:50 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:50.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:50 smithi149 ceph-mon[30657]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:50.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:50 smithi149 ceph-mon[30657]: Saving service prometheus spec with placement smithi149=a;count:1 2024-03-26T04:10:50.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:50 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:50 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:50 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:50 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.967 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:50 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:50.968 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:50 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:10:51.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:51 smithi204 ceph-mon[30652]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T04:10:51.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:51 smithi204 ceph-mon[30652]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T04:10:51.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:51 smithi204 ceph-mon[30652]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:10:51.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:51 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/2030943896' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:10:51.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:51 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:51.848 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled node-exporter update... 2024-03-26T04:10:51.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:51 smithi136 ceph-mon[25848]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T04:10:51.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:51 smithi136 ceph-mon[25848]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T04:10:51.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:51 smithi136 ceph-mon[25848]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:10:51.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:51 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/2030943896' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:10:51.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:51 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:51.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:51 smithi149 ceph-mon[30657]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T04:10:51.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:51 smithi149 ceph-mon[30657]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T04:10:51.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:51 smithi149 ceph-mon[30657]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:10:51.989 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:51 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/2030943896' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:10:51.989 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:51 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:52.522 DEBUG:teuthology.orchestra.run.smithi136:node-exporter.a> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T04:10:52.525 DEBUG:teuthology.orchestra.run.smithi149:node-exporter.b> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T04:10:52.528 DEBUG:teuthology.orchestra.run.smithi204:node-exporter.c> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@node-exporter.c.service 2024-03-26T04:10:52.530 INFO:tasks.cephadm:Adding alertmanager.a on smithi136 2024-03-26T04:10:52.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi136=a' 2024-03-26T04:10:52.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:52 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/2030943896' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:10:52.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:52 smithi204 ceph-mon[30652]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T04:10:52.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:52 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:52.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:52 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:52.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:52 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:52.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:52 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:52.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:52 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:52.803 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:10:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:52 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/2030943896' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:10:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:52 smithi136 ceph-mon[25848]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T04:10:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:52 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:52 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:52 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:52 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:52.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:52 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:52.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:52 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/2030943896' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:10:52.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:52 smithi149 ceph-mon[30657]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T04:10:52.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:52 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:52.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:52 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:52.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:52 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:52.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:52 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:52.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:52 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:53.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:53 smithi204 ceph-mon[30652]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi136=a;smithi149=b;smithi204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:53.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:53 smithi204 ceph-mon[30652]: Saving service node-exporter spec with placement smithi136=a;smithi149=b;smithi204=c;count:3 2024-03-26T04:10:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:53 smithi204 ceph-mon[30652]: Checking dashboard <-> RGW credentials 2024-03-26T04:10:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:53 smithi204 ceph-mon[30652]: pgmap v96: 33 pgs: 24 creating+peering, 8 unknown, 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T04:10:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:53 smithi204 ceph-mon[30652]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T04:10:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:53 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:10:53.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:53 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:10:53.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:53 smithi136 ceph-mon[25848]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi136=a;smithi149=b;smithi204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:53.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:53 smithi136 ceph-mon[25848]: Saving service node-exporter spec with placement smithi136=a;smithi149=b;smithi204=c;count:3 2024-03-26T04:10:53.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:53 smithi136 ceph-mon[25848]: Checking dashboard <-> RGW credentials 2024-03-26T04:10:53.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:53 smithi136 ceph-mon[25848]: pgmap v96: 33 pgs: 24 creating+peering, 8 unknown, 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T04:10:53.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:53 smithi136 ceph-mon[25848]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T04:10:53.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:53 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:10:53.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:53 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:10:53.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:53 smithi149 ceph-mon[30657]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi136=a;smithi149=b;smithi204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:53.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:53 smithi149 ceph-mon[30657]: Saving service node-exporter spec with placement smithi136=a;smithi149=b;smithi204=c;count:3 2024-03-26T04:10:53.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:53 smithi149 ceph-mon[30657]: Checking dashboard <-> RGW credentials 2024-03-26T04:10:53.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:53 smithi149 ceph-mon[30657]: pgmap v96: 33 pgs: 24 creating+peering, 8 unknown, 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T04:10:53.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:53 smithi149 ceph-mon[30657]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T04:10:53.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:53 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:10:53.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:53 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:10:54.466 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled alertmanager update... 2024-03-26T04:10:54.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:54 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:10:54.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:54 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:10:54.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:54 smithi204 ceph-mon[30652]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T04:10:54.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:54 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:54 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:10:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:54 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:10:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:54 smithi136 ceph-mon[25848]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T04:10:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:54 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:54.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:54 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:10:54.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:54 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:10:54.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:54 smithi149 ceph-mon[30657]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T04:10:54.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:54 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:54.992 DEBUG:teuthology.orchestra.run.smithi136:alertmanager.a> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T04:10:54.995 INFO:tasks.cephadm:Adding grafana.a on smithi149 2024-03-26T04:10:54.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi149=a' 2024-03-26T04:10:55.229 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:10:55.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:55 smithi204 ceph-mon[30652]: pgmap v99: 65 pgs: 32 unknown, 24 creating+peering, 9 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T04:10:55.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:55 smithi204 ceph-mon[30652]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:55.603 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:55 smithi204 ceph-mon[30652]: Saving service alertmanager spec with placement smithi136=a;count:1 2024-03-26T04:10:55.603 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:55 smithi204 ceph-mon[30652]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T04:10:55.603 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:55 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:10:55.603 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:55 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:10:55.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:55 smithi136 ceph-mon[25848]: pgmap v99: 65 pgs: 32 unknown, 24 creating+peering, 9 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T04:10:55.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:55 smithi136 ceph-mon[25848]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:55.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:55 smithi136 ceph-mon[25848]: Saving service alertmanager spec with placement smithi136=a;count:1 2024-03-26T04:10:55.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:55 smithi136 ceph-mon[25848]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T04:10:55.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:55 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:10:55.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:55 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:10:55.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:55 smithi149 ceph-mon[30657]: pgmap v99: 65 pgs: 32 unknown, 24 creating+peering, 9 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T04:10:55.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:55 smithi149 ceph-mon[30657]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:55.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:55 smithi149 ceph-mon[30657]: Saving service alertmanager spec with placement smithi136=a;count:1 2024-03-26T04:10:55.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:55 smithi149 ceph-mon[30657]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T04:10:55.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:55 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:10:55.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:55 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:10:56.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:56 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:10:56.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:56 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:10:56.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:56 smithi136 ceph-mon[25848]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T04:10:56.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:56 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:10:56.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:56 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:10:56.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:56 smithi149 ceph-mon[30657]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T04:10:56.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:56 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:10:56.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:56 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:10:56.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:56 smithi204 ceph-mon[30652]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T04:10:57.081 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled grafana update... 2024-03-26T04:10:57.686 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:57 smithi204 ceph-mon[30652]: pgmap v102: 97 pgs: 23 unknown, 9 creating+peering, 65 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T04:10:57.687 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:57 smithi204 ceph-mon[30652]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T04:10:57.687 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:57 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:10:57.687 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:57 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:10:57.687 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:57 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:57.713 DEBUG:teuthology.orchestra.run.smithi149:grafana.a> sudo journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T04:10:57.716 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T04:10:57.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:10:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:57 smithi136 ceph-mon[25848]: pgmap v102: 97 pgs: 23 unknown, 9 creating+peering, 65 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T04:10:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:57 smithi136 ceph-mon[25848]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T04:10:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:57 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:10:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:57 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:10:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:57 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:57.962 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:57.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:57 smithi149 ceph-mon[30657]: pgmap v102: 97 pgs: 23 unknown, 9 creating+peering, 65 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T04:10:57.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:57 smithi149 ceph-mon[30657]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T04:10:57.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:57 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:10:57.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:57 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:10:57.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:57 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:58.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:58 smithi136 ceph-mon[25848]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:58.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:58 smithi136 ceph-mon[25848]: Saving service grafana spec with placement smithi149=a;count:1 2024-03-26T04:10:58.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:58 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:10:58.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:58 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:10:58.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:58 smithi136 ceph-mon[25848]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T04:10:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:58 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:10:58.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:58 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:10:58.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:58 smithi204 ceph-mon[30652]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:58.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:58 smithi204 ceph-mon[30652]: Saving service grafana spec with placement smithi149=a;count:1 2024-03-26T04:10:58.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:58 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:10:58.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:58 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:10:58.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:58 smithi204 ceph-mon[30652]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T04:10:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:58 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:10:58.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:58 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:10:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:58 smithi149 ceph-mon[30657]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:58 smithi149 ceph-mon[30657]: Saving service grafana spec with placement smithi149=a;count:1 2024-03-26T04:10:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:58 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:10:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:58 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:10:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:58 smithi149 ceph-mon[30657]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T04:10:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:58 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:10:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:58 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:10:59.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:59 smithi136 ceph-mon[25848]: pgmap v105: 129 pgs: 22 unknown, 19 creating+peering, 88 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-26T04:10:59.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:59 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:10:59.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:59 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:10:59.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:59 smithi136 ceph-mon[25848]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T04:10:59.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:59 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:59.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:59 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:59.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:10:59 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:59.940 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-03-26T04:10:59.941 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDTSgJmpCQAOBAAKtKFwrpW/HhkQa+Qq7MdLw== 2024-03-26T04:10:59.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:59 smithi149 ceph-mon[30657]: pgmap v105: 129 pgs: 22 unknown, 19 creating+peering, 88 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-26T04:10:59.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:59 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:10:59.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:59 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:10:59.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:59 smithi149 ceph-mon[30657]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T04:10:59.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:59 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:59.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:59 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:59.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:10:59 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:59.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:59 smithi204 ceph-mon[30652]: pgmap v105: 129 pgs: 22 unknown, 19 creating+peering, 88 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-26T04:10:59.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:59 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/4232082647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:10:59.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:59 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/643392347' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:10:59.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:59 smithi204 ceph-mon[30652]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T04:10:59.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:59 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:59.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:59 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:10:59.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:10:59 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:11:00.531 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:11:00.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T04:11:00.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T04:11:00.575 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:11:00.737 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:00 smithi149 ceph-mon[30657]: Deploying daemon prometheus.a on smithi149 2024-03-26T04:11:00.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:00 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/4233415784' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:00.738 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:00 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/4233415784' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:11:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:00 smithi136 ceph-mon[25848]: Deploying daemon prometheus.a on smithi149 2024-03-26T04:11:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:00 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4233415784' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:00 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/4233415784' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:11:00.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:00 smithi204 ceph-mon[30652]: Deploying daemon prometheus.a on smithi149 2024-03-26T04:11:00.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:00 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/4233415784' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:00.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:00 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/4233415784' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:11:01.254 INFO:teuthology.orchestra.run.smithi149.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:11:01.696 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:01 smithi149 ceph-mon[30657]: pgmap v107: 129 pgs: 22 unknown, 19 creating+peering, 88 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-03-26T04:11:01.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:01 smithi136 ceph-mon[25848]: pgmap v107: 129 pgs: 22 unknown, 19 creating+peering, 88 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-03-26T04:11:01.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:01 smithi204 ceph-mon[30652]: pgmap v107: 129 pgs: 22 unknown, 19 creating+peering, 88 active+clean; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-03-26T04:11:03.845 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:03 smithi149 ceph-mon[30657]: pgmap v108: 129 pgs: 10 creating+peering, 119 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 4.0 KiB/s wr, 168 op/s 2024-03-26T04:11:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:03 smithi136 ceph-mon[25848]: pgmap v108: 129 pgs: 10 creating+peering, 119 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 4.0 KiB/s wr, 168 op/s 2024-03-26T04:11:03.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:03 smithi204 ceph-mon[30652]: pgmap v108: 129 pgs: 10 creating+peering, 119 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 4.0 KiB/s wr, 168 op/s 2024-03-26T04:11:04.759 INFO:teuthology.orchestra.run.smithi149.stdout:[client.1] 2024-03-26T04:11:04.760 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDYSgJm9/sSLRAA8hHUZm6okGx1eFBbVzfxAw== 2024-03-26T04:11:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:05 smithi136 ceph-mon[25848]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 91 KiB/s rd, 4.6 KiB/s wr, 203 op/s 2024-03-26T04:11:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:05 smithi136 ceph-mon[25848]: from='client.? 172.21.15.149:0/2929580328' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:05 smithi136 ceph-mon[25848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:05 smithi136 ceph-mon[25848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:11:05.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:05 smithi149 ceph-mon[30657]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 91 KiB/s rd, 4.6 KiB/s wr, 203 op/s 2024-03-26T04:11:05.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:05 smithi149 ceph-mon[30657]: from='client.? 172.21.15.149:0/2929580328' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:05.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:05 smithi149 ceph-mon[30657]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:05.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:05 smithi149 ceph-mon[30657]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:11:05.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:05 smithi204 ceph-mon[30652]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 91 KiB/s rd, 4.6 KiB/s wr, 203 op/s 2024-03-26T04:11:05.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:05 smithi204 ceph-mon[30652]: from='client.? 172.21.15.149:0/2929580328' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:05.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:05 smithi204 ceph-mon[30652]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:05.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:05 smithi204 ceph-mon[30652]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:11:06.084 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:11:06.084 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T04:11:06.084 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T04:11:06.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:11:06.371 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T04:11:07.730 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:07 smithi149 ceph-mon[30657]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 3.9 KiB/s wr, 180 op/s 2024-03-26T04:11:07.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:07 smithi136 ceph-mon[25848]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 3.9 KiB/s wr, 180 op/s 2024-03-26T04:11:07.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:07 smithi204 ceph-mon[30652]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 3.9 KiB/s wr, 180 op/s 2024-03-26T04:11:08.037 INFO:teuthology.orchestra.run.smithi204.stdout:[client.2] 2024-03-26T04:11:08.037 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDcSgJmrMD5ARAAn7XKwSZYDho21gJFQQrdkA== 2024-03-26T04:11:08.598 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:11:08.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-26T04:11:08.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-26T04:11:08.623 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:08 smithi204 ceph-mon[30652]: from='client.? 172.21.15.204:0/2679278134' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:08.623 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:08 smithi204 ceph-mon[30652]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:08.624 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:08 smithi204 ceph-mon[30652]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:11:08.640 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T04:11:08.640 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T04:11:08.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T04:11:08.883 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:08.886 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:08 smithi149 ceph-mon[30657]: from='client.? 172.21.15.204:0/2679278134' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:08.886 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:08 smithi149 ceph-mon[30657]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:08.886 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:08 smithi149 ceph-mon[30657]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:11:08.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:08 smithi136 ceph-mon[25848]: from='client.? 172.21.15.204:0/2679278134' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:08.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:08 smithi136 ceph-mon[25848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:11:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:08 smithi136 ceph-mon[25848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:11:09.488 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 systemd[1]: Starting Ceph prometheus.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:11:09.816 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-mon[30657]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 72 KiB/s rd, 3.5 KiB/s wr, 160 op/s 2024-03-26T04:11:09.816 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 podman[41090]: 2024-03-26 04:11:09.457243692 +0000 UTC m=+0.018645022 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T04:11:09.816 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 podman[41090]: 2024-03-26 04:11:09.580911115 +0000 UTC m=+0.142312420 container create 669c70d350055e7f287863846c1cd58eb12a6c43a1cbea655e255a1e40b24a01 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:11:09.816 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 podman[41090]: 2024-03-26 04:11:09.81633149 +0000 UTC m=+0.377732801 container init 669c70d350055e7f287863846c1cd58eb12a6c43a1cbea655e255a1e40b24a01 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:11:09.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:09 smithi136 ceph-mon[25848]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 72 KiB/s rd, 3.5 KiB/s wr, 160 op/s 2024-03-26T04:11:09.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:09 smithi204 ceph-mon[30652]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 72 KiB/s rd, 3.5 KiB/s wr, 160 op/s 2024-03-26T04:11:10.238 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 podman[41090]: 2024-03-26 04:11:09.821708509 +0000 UTC m=+0.383109819 container start 669c70d350055e7f287863846c1cd58eb12a6c43a1cbea655e255a1e40b24a01 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:11:10.238 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.845Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T04:11:10.238 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.845Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.845Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi149 (none))" 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.845Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.845Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.846Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.847Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.848Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.848Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.851Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.851Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.718µs 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.851Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.851Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T04:11:10.239 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.851Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.026µs wal_replay_duration=209.853µs wbl_replay_duration=424ns total_replay_duration=245.056µs 2024-03-26T04:11:10.240 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.852Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T04:11:10.240 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.852Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T04:11:10.240 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.852Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T04:11:10.240 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.870Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.231429ms db_storage=1.084µs remote_storage=1.143µs web_handler=351ns query_engine=499ns scrape=402.785µs scrape_sd=85.2µs notify=747ns notify_sd=1.411µs rules=17.562579ms tracing=6.62µs 2024-03-26T04:11:10.240 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.870Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T04:11:10.240 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:11:09.870Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T04:11:10.240 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 bash[41090]: 669c70d350055e7f287863846c1cd58eb12a6c43a1cbea655e255a1e40b24a01 2024-03-26T04:11:10.240 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:11:09 smithi149 systemd[1]: Started Ceph prometheus.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:11:10.976 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:11:11.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:11:11.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:11:11.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:11:11.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:10 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:11:11.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:10 smithi149 ceph-mon[30657]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.9 KiB/s wr, 134 op/s 2024-03-26T04:11:11.238 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:11 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: ignoring --setuser ceph since I am not root 2024-03-26T04:11:11.238 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:11 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: ignoring --setgroup ceph since I am not root 2024-03-26T04:11:11.238 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:11 smithi149 ceph-mgr[32137]: -- 172.21.15.149:0/2608212221 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b30939c1a0 con 0x55b309372800 2024-03-26T04:11:11.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:11:11.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:11:11.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:11:11.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:10 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:11:11.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:10 smithi204 ceph-mon[30652]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.9 KiB/s wr, 134 op/s 2024-03-26T04:11:11.248 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:11 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: ignoring --setuser ceph since I am not root 2024-03-26T04:11:11.248 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:11 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: ignoring --setgroup ceph since I am not root 2024-03-26T04:11:11.248 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:11 smithi204 ceph-mgr[32759]: -- 172.21.15.204:0/4187187927 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560cb1c0e1a0 con 0x560cb1be4c00 2024-03-26T04:11:11.295 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:11:11.295 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:11:11.296 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' 2024-03-26T04:11:11.296 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:10 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:11:11.296 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:10 smithi136 ceph-mon[25848]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.9 KiB/s wr, 134 op/s 2024-03-26T04:11:11.296 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:11 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ignoring --setuser ceph since I am not root 2024-03-26T04:11:11.296 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:11 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ignoring --setgroup ceph since I am not root 2024-03-26T04:11:11.698 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2883767701},{"type":"v1","addr":"172.21.15.136:6801","nonce":2883767701}]},"active_addr":"172.21.15.136:6801/2883767701","active_change":"2024-03-26T04:08:06.426032+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.136:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":2214712787}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":2002033214}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":1581823917}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":1002612031}]}]} 2024-03-26T04:11:11.704 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T04:11:11.704 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T04:11:11.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:11:11.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:11 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:11.606+0000 7fea5dd06200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:11:11.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:11 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:11.706+0000 7fea5dd06200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:11:11.957 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:11.965 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:11 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:11.581+0000 7fe3aea79200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:11:11.965 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:11 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:11.673+0000 7fe3aea79200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:11:11.965 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:11 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:11.572+0000 7f47a2a9c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:11:11.965 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:11 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:11.664+0000 7f47a2a9c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:11:12.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:11 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/2515936254' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:11:12.222 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:11 smithi136 ceph-mon[25848]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:11:12.222 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:11 smithi136 ceph-mon[25848]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T04:11:12.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:11 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/2515936254' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:11:12.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:11 smithi149 ceph-mon[30657]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:11:12.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:11 smithi149 ceph-mon[30657]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T04:11:12.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:11 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/2515936254' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:11:12.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:11 smithi204 ceph-mon[30652]: from='mgr.14152 172.21.15.136:0/1545613401' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:11:12.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:11 smithi204 ceph-mon[30652]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T04:11:12.486 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:12 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:12.385+0000 7fea5dd06200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:11:12.616 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:12 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:12.293+0000 7f47a2a9c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:11:12.616 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:12 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:12.384+0000 7f47a2a9c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:11:12.616 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:12 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:12.523+0000 7f47a2a9c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:11:12.616 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:12 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:12.616+0000 7f47a2a9c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:11:12.642 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:12 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:12.312+0000 7fe3aea79200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:11:12.642 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:12 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:12.403+0000 7fe3aea79200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:11:12.642 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:12 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:12.545+0000 7fe3aea79200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:11:12.642 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:12 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:12.640+0000 7fe3aea79200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:11:12.811 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:12 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:12.486+0000 7fea5dd06200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:11:12.811 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:12 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:12.629+0000 7fea5dd06200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:11:12.811 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:12 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:12.725+0000 7fea5dd06200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:11:12.811 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:12 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:12.810+0000 7fea5dd06200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:11:12.959 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:12 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:12.698+0000 7f47a2a9c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:11:12.960 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:12 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:12.959+0000 7f47a2a9c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:11:12.987 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:12 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:12.723+0000 7fe3aea79200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:11:13.246 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:13 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:13.067+0000 7f47a2a9c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:11:13.246 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:13 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:13.152+0000 7f47a2a9c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:11:13.246 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:13 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:13.245+0000 7f47a2a9c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:11:13.277 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:12 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:12.989+0000 7fe3aea79200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:11:13.277 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:13.096+0000 7fe3aea79200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:11:13.277 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:13.181+0000 7fe3aea79200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:11:13.277 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:13.276+0000 7fe3aea79200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:11:13.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:13 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:13.078+0000 7fea5dd06200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:11:13.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:13 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:13.198+0000 7fea5dd06200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:11:13.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:13 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:13.286+0000 7fea5dd06200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:11:13.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:13 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:13.382+0000 7fea5dd06200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:11:13.497 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:13 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:13.420+0000 7f47a2a9c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:11:13.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:13 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:13.568+0000 7fea5dd06200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:11:13.737 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:13.455+0000 7fe3aea79200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:11:13.837 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:11:13.837 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":30,"fsid":"276a1e6c-eb26-11ee-95d0-87774f69a715","created":"2024-03-26T04:07:10.618462+0000","modified":"2024-03-26T04:10:58.579514+0000","last_up_change":"2024-03-26T04:10:35.047970+0000","last_in_change":"2024-03-26T04:10:16.269354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:10:36.777571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T04:10:50.138450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-26T04:10:51.841351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-26T04:10:53.796811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-26T04:10:55.855198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8bc9716b-db6c-4d3a-a77b-0424e38088a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6803","nonce":4198578744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6805","nonce":4198578744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6809","nonce":4198578744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6807","nonce":4198578744}]},"public_addr":"172.21.15.136:6803/4198578744","cluster_addr":"172.21.15.136:6805/4198578744","heartbeat_back_addr":"172.21.15.136:6809/4198578744","heartbeat_front_addr":"172.21.15.136:6807/4198578744","state":["exists","up"]},{"osd":1,"uuid":"b00b6ef7-9fd4-402b-a04d-5db2fea2d01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6801","nonce":883566559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6803","nonce":883566559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6807","nonce":883566559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6805","nonce":883566559}]},"public_addr":"172.21.15.149:6801/883566559","cluster_addr":"172.21.15.149:6803/883566559","heartbeat_back_addr":"172.21.15.149:6807/883566559","heartbeat_front_addr":"172.21.15.149:6805/883566559","state":["exists","up"]},{"osd":2,"uuid":"faeabeb6-c22b-4832-ae19-1e44f9524e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6801","nonce":3457499621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6803","nonce":3457499621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6807","nonce":3457499621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6805","nonce":3457499621}]},"public_addr":"172.21.15.204:6801/3457499621","cluster_addr":"172.21.15.204:6803/3457499621","heartbeat_back_addr":"172.21.15.204:6807/3457499621","heartbeat_front_addr":"172.21.15.204:6805/3457499621","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:09:44.740827+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:10:08.766950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:10:32.825143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:6801/3121969854":"2024-03-27T04:08:06.425886+0000","172.21.15.136:6800/3121969854":"2024-03-27T04:08:06.425886+0000","172.21.15.136:0/2374692480":"2024-03-27T04:08:06.425886+0000","172.21.15.136:0/2817395307":"2024-03-27T04:07:39.177832+0000","172.21.15.136:0/3540036637":"2024-03-27T04:08:06.425886+0000","172.21.15.136:6801/391737807":"2024-03-27T04:07:39.177832+0000","172.21.15.136:0/3458371792":"2024-03-27T04:07:39.177832+0000","172.21.15.136:0/4153123335":"2024-03-27T04:08:06.425886+0000","172.21.15.136:0/1442132325":"2024-03-27T04:07:39.177832+0000","172.21.15.136:6800/391737807":"2024-03-27T04:07:39.177832+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:11:14.166 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:13 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1730077284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:11:14.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:13 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/1730077284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:11:14.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:13 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/1730077284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:11:14.485 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T04:11:14.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:11:14.725 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:14.737 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:14 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:14.427+0000 7fe3aea79200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:11:14.738 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:14 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:14.558+0000 7fe3aea79200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:11:14.747 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:14 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:14.374+0000 7f47a2a9c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:11:14.747 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:14 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:14.502+0000 7f47a2a9c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:11:14.847 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:14 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:14.561+0000 7fea5dd06200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:11:14.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:14 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:14.700+0000 7fea5dd06200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:11:15.114 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:14 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:14.832+0000 7f47a2a9c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:11:15.114 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:14 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:14.921+0000 7f47a2a9c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:11:15.114 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:15 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:15.113+0000 7f47a2a9c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:11:15.151 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:15 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:15.053+0000 7fea5dd06200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:11:15.151 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:15 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:15.150+0000 7fea5dd06200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:11:15.184 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:14 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:14.897+0000 7fe3aea79200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:11:15.184 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:14 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:14.986+0000 7fe3aea79200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:11:15.184 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:15 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:15.183+0000 7fe3aea79200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:11:15.448 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:15 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:15.348+0000 7fea5dd06200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:11:15.487 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:15 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:15.368+0000 7fe3aea79200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:11:15.497 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:15 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:15.296+0000 7f47a2a9c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:11:15.796 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:15 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:15.536+0000 7fea5dd06200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:11:15.835 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:15 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:15.577+0000 7f47a2a9c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:11:15.835 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:15 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:15.835+0000 7f47a2a9c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:11:15.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:15 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:15.654+0000 7fe3aea79200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:11:16.072 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:15 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:15.827+0000 7fea5dd06200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:11:16.237 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:15 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:15.921+0000 7fe3aea79200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:11:16.238 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:16 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:16.158+0000 7fe3aea79200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:11:16.247 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:16 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:16.068+0000 7f47a2a9c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:11:16.344 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:16 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:16.099+0000 7fea5dd06200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:11:16.345 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:16 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:16.344+0000 7fea5dd06200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:11:16.523 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:11:16.523 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":30,"fsid":"276a1e6c-eb26-11ee-95d0-87774f69a715","created":"2024-03-26T04:07:10.618462+0000","modified":"2024-03-26T04:10:58.579514+0000","last_up_change":"2024-03-26T04:10:35.047970+0000","last_in_change":"2024-03-26T04:10:16.269354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:10:36.777571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T04:10:50.138450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-26T04:10:51.841351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-26T04:10:53.796811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-26T04:10:55.855198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8bc9716b-db6c-4d3a-a77b-0424e38088a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6803","nonce":4198578744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6805","nonce":4198578744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6809","nonce":4198578744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":4198578744},{"type":"v1","addr":"172.21.15.136:6807","nonce":4198578744}]},"public_addr":"172.21.15.136:6803/4198578744","cluster_addr":"172.21.15.136:6805/4198578744","heartbeat_back_addr":"172.21.15.136:6809/4198578744","heartbeat_front_addr":"172.21.15.136:6807/4198578744","state":["exists","up"]},{"osd":1,"uuid":"b00b6ef7-9fd4-402b-a04d-5db2fea2d01c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6801","nonce":883566559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6803","nonce":883566559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6807","nonce":883566559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":883566559},{"type":"v1","addr":"172.21.15.149:6805","nonce":883566559}]},"public_addr":"172.21.15.149:6801/883566559","cluster_addr":"172.21.15.149:6803/883566559","heartbeat_back_addr":"172.21.15.149:6807/883566559","heartbeat_front_addr":"172.21.15.149:6805/883566559","state":["exists","up"]},{"osd":2,"uuid":"faeabeb6-c22b-4832-ae19-1e44f9524e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6801","nonce":3457499621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6803","nonce":3457499621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6807","nonce":3457499621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3457499621},{"type":"v1","addr":"172.21.15.204:6805","nonce":3457499621}]},"public_addr":"172.21.15.204:6801/3457499621","cluster_addr":"172.21.15.204:6803/3457499621","heartbeat_back_addr":"172.21.15.204:6807/3457499621","heartbeat_front_addr":"172.21.15.204:6805/3457499621","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:09:44.740827+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:10:08.766950+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:10:32.825143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:6801/3121969854":"2024-03-27T04:08:06.425886+0000","172.21.15.136:6800/3121969854":"2024-03-27T04:08:06.425886+0000","172.21.15.136:0/2374692480":"2024-03-27T04:08:06.425886+0000","172.21.15.136:0/2817395307":"2024-03-27T04:07:39.177832+0000","172.21.15.136:0/3540036637":"2024-03-27T04:08:06.425886+0000","172.21.15.136:6801/391737807":"2024-03-27T04:07:39.177832+0000","172.21.15.136:0/3458371792":"2024-03-27T04:07:39.177832+0000","172.21.15.136:0/4153123335":"2024-03-27T04:08:06.425886+0000","172.21.15.136:0/1442132325":"2024-03-27T04:07:39.177832+0000","172.21.15.136:6800/391737807":"2024-03-27T04:07:39.177832+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:11:16.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:16 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/18480084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:11:16.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:16 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/18480084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:11:16.968 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:16 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/18480084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:11:17.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T04:11:17.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T04:11:17.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T04:11:17.237 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:16 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:16.967+0000 7fe3aea79200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:11:17.247 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:16 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:16.859+0000 7f47a2a9c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:11:17.386 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:17.391 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:17.401 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:17.448 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:17 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:17.189+0000 7fea5dd06200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:11:17.533 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:17.431+0000 7f47a2a9c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:11:17.533 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:17.532+0000 7f47a2a9c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:11:17.844 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:17.760+0000 7f47a2a9c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:11:17.844 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: [26/Mar/2024:04:11:17] ENGINE Bus STARTING 2024-03-26T04:11:17.844 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:17.544+0000 7fe3aea79200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:11:17.844 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:17.644+0000 7fe3aea79200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:11:18.117 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:17 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:17.809+0000 7fea5dd06200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:11:18.118 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:17 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:17.916+0000 7fea5dd06200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:11:18.118 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:17 smithi136 ceph-mon[25848]: Standby manager daemon z restarted 2024-03-26T04:11:18.118 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:17 smithi136 ceph-mon[25848]: Standby manager daemon z started 2024-03-26T04:11:18.118 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:17 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:11:18.118 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:17 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:11:18.118 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:17 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:11:18.118 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:17 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:11:18.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-mon[30657]: Standby manager daemon z restarted 2024-03-26T04:11:18.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-mon[30657]: Standby manager daemon z started 2024-03-26T04:11:18.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:11:18.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:11:18.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:11:18.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:11:18.238 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:17.865+0000 7fe3aea79200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:11:18.238 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: [26/Mar/2024:04:11:17] ENGINE Bus STARTING 2024-03-26T04:11:18.239 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: CherryPy Checker: 2024-03-26T04:11:18.239 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: The Application mounted at '' has an empty config. 2024-03-26T04:11:18.239 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: 2024-03-26T04:11:18.239 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: [26/Mar/2024:04:11:17] ENGINE Serving on http://:::9283 2024-03-26T04:11:18.239 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:11:17 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x[32114]: [26/Mar/2024:04:11:17] ENGINE Bus STARTED 2024-03-26T04:11:18.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-mon[30652]: Standby manager daemon z restarted 2024-03-26T04:11:18.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-mon[30652]: Standby manager daemon z started 2024-03-26T04:11:18.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T04:11:18.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:11:18.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T04:11:18.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.204:0/1534017226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:11:18.248 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: CherryPy Checker: 2024-03-26T04:11:18.248 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: The Application mounted at '' has an empty config. 2024-03-26T04:11:18.248 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: 2024-03-26T04:11:18.248 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: [26/Mar/2024:04:11:17] ENGINE Serving on http://:::9283 2024-03-26T04:11:18.248 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:11:17 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z[32736]: [26/Mar/2024:04:11:17] ENGINE Bus STARTED 2024-03-26T04:11:18.379 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:18.141+0000 7fea5dd06200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:11:18.379 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:11:18] ENGINE Bus STARTING 2024-03-26T04:11:18.633 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: CherryPy Checker: 2024-03-26T04:11:18.634 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: The Application mounted at '' has an empty config. 2024-03-26T04:11:18.634 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:11:18.634 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:11:18] ENGINE Serving on http://:::9283 2024-03-26T04:11:18.634 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:11:18] ENGINE Bus STARTED 2024-03-26T04:11:18.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: Standby manager daemon x restarted 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: Standby manager daemon x started 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: Active manager daemon y restarted 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: Activating manager daemon y 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: mgrmap e19: y(active, starting, since 0.0118141s), standbys: z, x 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:11:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: Manager daemon y is now available 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:18.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:11:18.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:18 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:11:19.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-26T04:11:19.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: Standby manager daemon x restarted 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: Standby manager daemon x started 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: Active manager daemon y restarted 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: Activating manager daemon y 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: mgrmap e19: y(active, starting, since 0.0118141s), standbys: z, x 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:11:19.239 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: Manager daemon y is now available 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:11:19.240 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:18 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:11:19.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-26T04:11:19.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: Standby manager daemon x restarted 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: Standby manager daemon x started 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.? 172.21.15.149:0/719218572' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: Active manager daemon y restarted 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: Activating manager daemon y 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: mgrmap e19: y(active, starting, since 0.0118141s), standbys: z, x 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:11:19.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: Manager daemon y is now available 2024-03-26T04:11:19.249 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:11:19.250 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:11:19.250 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:19.250 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:19.250 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:11:19.250 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:18 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:11:19.449 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:19 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ::ffff:172.21.15.149 - - [26/Mar/2024:04:11:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:19.847 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:19 smithi149 ceph-mon[30657]: [26/Mar/2024:04:11:18] ENGINE Bus STARTING 2024-03-26T04:11:19.847 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:19 smithi149 ceph-mon[30657]: [26/Mar/2024:04:11:18] ENGINE Serving on https://172.21.15.136:7150 2024-03-26T04:11:19.847 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:19 smithi149 ceph-mon[30657]: [26/Mar/2024:04:11:18] ENGINE Serving on http://172.21.15.136:8765 2024-03-26T04:11:19.847 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:19 smithi149 ceph-mon[30657]: [26/Mar/2024:04:11:18] ENGINE Bus STARTED 2024-03-26T04:11:19.848 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:19 smithi149 ceph-mon[30657]: mgrmap e20: y(active, since 1.01604s), standbys: z, x 2024-03-26T04:11:20.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:19 smithi136 ceph-mon[25848]: [26/Mar/2024:04:11:18] ENGINE Bus STARTING 2024-03-26T04:11:20.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:19 smithi136 ceph-mon[25848]: [26/Mar/2024:04:11:18] ENGINE Serving on https://172.21.15.136:7150 2024-03-26T04:11:20.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:19 smithi136 ceph-mon[25848]: [26/Mar/2024:04:11:18] ENGINE Serving on http://172.21.15.136:8765 2024-03-26T04:11:20.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:19 smithi136 ceph-mon[25848]: [26/Mar/2024:04:11:18] ENGINE Bus STARTED 2024-03-26T04:11:20.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:19 smithi136 ceph-mon[25848]: mgrmap e20: y(active, since 1.01604s), standbys: z, x 2024-03-26T04:11:20.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:19 smithi204 ceph-mon[30652]: [26/Mar/2024:04:11:18] ENGINE Bus STARTING 2024-03-26T04:11:20.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:19 smithi204 ceph-mon[30652]: [26/Mar/2024:04:11:18] ENGINE Serving on https://172.21.15.136:7150 2024-03-26T04:11:20.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:19 smithi204 ceph-mon[30652]: [26/Mar/2024:04:11:18] ENGINE Serving on http://172.21.15.136:8765 2024-03-26T04:11:20.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:19 smithi204 ceph-mon[30652]: [26/Mar/2024:04:11:18] ENGINE Bus STARTED 2024-03-26T04:11:20.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:19 smithi204 ceph-mon[30652]: mgrmap e20: y(active, since 1.01604s), standbys: z, x 2024-03-26T04:11:21.137 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:20 smithi136 ceph-mon[25848]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:21.138 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:20 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.138 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:20 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.138 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:20 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.138 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:20 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:20 smithi149 ceph-mon[30657]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:20 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:20 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:20 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:20 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:20 smithi204 ceph-mon[30652]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:21.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:20 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:20 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:20 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:21.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:20 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:21 smithi136 ceph-mon[25848]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:22.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:21 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:21 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:21 smithi136 ceph-mon[25848]: mgrmap e21: y(active, since 3s), standbys: z, x 2024-03-26T04:11:22.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:21 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:21 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:21 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:11:22.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:21 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:21 smithi149 ceph-mon[30657]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:22.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:21 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:21 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:21 smithi149 ceph-mon[30657]: mgrmap e21: y(active, since 3s), standbys: z, x 2024-03-26T04:11:22.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:21 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:21 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:21 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:11:22.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:21 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:21 smithi204 ceph-mon[30652]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:22.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:21 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:21 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:21 smithi204 ceph-mon[30652]: mgrmap e21: y(active, since 3s), standbys: z, x 2024-03-26T04:11:22.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:21 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:21 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:21 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:11:22.248 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:21 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:22.516 INFO:teuthology.orchestra.run.smithi136.stdout:34359738389 2024-03-26T04:11:22.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T04:11:23.117 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:22 smithi136 ceph-mon[25848]: Adjusting osd_memory_target on smithi149 to 4792M 2024-03-26T04:11:23.143 INFO:teuthology.orchestra.run.smithi136.stdout:77309411340 2024-03-26T04:11:23.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T04:11:23.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:22 smithi149 ceph-mon[30657]: Adjusting osd_memory_target on smithi149 to 4792M 2024-03-26T04:11:23.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:22 smithi204 ceph-mon[30652]: Adjusting osd_memory_target on smithi149 to 4792M 2024-03-26T04:11:23.411 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:24.054 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:23 smithi136 ceph-mon[25848]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:24.055 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:23 smithi136 ceph-mon[25848]: mgrmap e22: y(active, since 4s), standbys: z, x 2024-03-26T04:11:24.081 INFO:teuthology.orchestra.run.smithi136.stdout:55834574864 2024-03-26T04:11:24.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T04:11:24.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:23 smithi149 ceph-mon[30657]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:24.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:23 smithi149 ceph-mon[30657]: mgrmap e22: y(active, since 4s), standbys: z, x 2024-03-26T04:11:24.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:23 smithi204 ceph-mon[30652]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:24.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:23 smithi204 ceph-mon[30652]: mgrmap e22: y(active, since 4s), standbys: z, x 2024-03-26T04:11:24.613 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:26.053 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:26.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:25 smithi149 ceph-mon[30657]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-26T04:11:26.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:25 smithi204 ceph-mon[30652]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-26T04:11:26.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:25 smithi136 ceph-mon[25848]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-26T04:11:28.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:27 smithi136 ceph-mon[25848]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T04:11:28.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:27 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:28.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:27 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:28.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:27 smithi149 ceph-mon[30657]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T04:11:28.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:27 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:28.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:27 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:28.239 INFO:teuthology.orchestra.run.smithi136.stdout:34359738390 2024-03-26T04:11:28.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:27 smithi204 ceph-mon[30652]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T04:11:28.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:27 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:28.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:27 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:28.666 INFO:teuthology.orchestra.run.smithi136.stdout:77309411341 2024-03-26T04:11:29.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:28 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/1059349137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:11:29.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:28 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/329169527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:11:29.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:28 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/1059349137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:11:29.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:28 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/329169527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:11:29.288 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:28 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/1059349137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:11:29.288 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:28 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/329169527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:11:29.288 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:29 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ::ffff:172.21.15.149 - - [26/Mar/2024:04:11:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:29.314 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411340 got 77309411341 for osd.2 2024-03-26T04:11:29.314 DEBUG:teuthology.parallel:result is None 2024-03-26T04:11:30.074 INFO:teuthology.orchestra.run.smithi136.stdout:55834574865 2024-03-26T04:11:30.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:29 smithi136 ceph-mon[25848]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:11:30.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:29 smithi149 ceph-mon[30657]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:11:30.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:29 smithi204 ceph-mon[30652]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:11:30.551 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738389 got 34359738390 for osd.0 2024-03-26T04:11:30.552 DEBUG:teuthology.parallel:result is None 2024-03-26T04:11:31.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:30 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/722599841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:11:31.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:30 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:31.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:30 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:31.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:30 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:11:31.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:30 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:31.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:30 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:11:31.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:30 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:11:31.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:30 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/722599841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:11:31.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:30 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:31.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:30 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:31.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:30 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:11:31.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:30 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:31.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:30 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:11:31.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:30 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:11:31.321 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:30 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/722599841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:11:31.321 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:30 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:31.321 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:30 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:31.321 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:30 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:11:31.321 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:30 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:31.321 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:30 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:11:31.321 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:30 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:11:31.355 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574864 got 55834574865 for osd.1 2024-03-26T04:11:31.355 DEBUG:teuthology.parallel:result is None 2024-03-26T04:11:31.355 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T04:11:31.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:11:31.592 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Adjusting osd_memory_target on smithi136 to 16064M 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:32.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:32.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:11:32.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:11:32.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:11:32.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:32 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Adjusting osd_memory_target on smithi136 to 16064M 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:11:32.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:11:32.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:11:32.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:32 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Adjusting osd_memory_target on smithi136 to 16064M 2024-03-26T04:11:32.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T04:11:32.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi149:/etc/ceph/ceph.conf 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: Updating smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:32.499 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:32 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:33.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:33 smithi136 ceph-mon[25848]: Deploying daemon alertmanager.a on smithi136 2024-03-26T04:11:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:11:33.461 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-03-26T04:11:33.487 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:33 smithi149 ceph-mon[30657]: Deploying daemon alertmanager.a on smithi136 2024-03-26T04:11:33.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:33 smithi204 ceph-mon[30652]: Deploying daemon alertmanager.a on smithi136 2024-03-26T04:11:34.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:34 smithi136 ceph-mon[25848]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T04:11:34.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:34 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:34.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:34 smithi149 ceph-mon[30657]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T04:11:34.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:34 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:34.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:34 smithi204 ceph-mon[30652]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T04:11:34.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:34 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:34.616 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-26T04:11:32.175004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96624,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126544,"statfs":{"total":287972524032,"available":287873581056,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002537"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162609+0000","last_change":"2024-03-26T04:10:57.593375+0000","last_active":"2024-03-26T04:11:18.162609+0000","last_peered":"2024-03-26T04:11:18.162609+0000","last_clean":"2024-03-26T04:11:18.162609+0000","last_became_active":"2024-03-26T04:10:57.592779+0000","last_became_peered":"2024-03-26T04:10:57.592779+0000","last_unstale":"2024-03-26T04:11:18.162609+0000","last_undegraded":"2024-03-26T04:11:18.162609+0000","last_fullsized":"2024-03-26T04:11:18.162609+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:33:12.453911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.161923+0000","last_change":"2024-03-26T04:10:55.570493+0000","last_active":"2024-03-26T04:11:18.161923+0000","last_peered":"2024-03-26T04:11:18.161923+0000","last_clean":"2024-03-26T04:11:18.161923+0000","last_became_active":"2024-03-26T04:10:55.570031+0000","last_became_peered":"2024-03-26T04:10:55.570031+0000","last_unstale":"2024-03-26T04:11:18.161923+0000","last_undegraded":"2024-03-26T04:11:18.161923+0000","last_fullsized":"2024-03-26T04:11:18.161923+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:16:20.291852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250919+0000","last_change":"2024-03-26T04:10:51.541831+0000","last_active":"2024-03-26T04:11:18.250919+0000","last_peered":"2024-03-26T04:11:18.250919+0000","last_clean":"2024-03-26T04:11:18.250919+0000","last_became_active":"2024-03-26T04:10:51.541118+0000","last_became_peered":"2024-03-26T04:10:51.541118+0000","last_unstale":"2024-03-26T04:11:18.250919+0000","last_undegraded":"2024-03-26T04:11:18.250919+0000","last_fullsized":"2024-03-26T04:11:18.250919+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:39:42.843470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250994+0000","last_change":"2024-03-26T04:10:53.561664+0000","last_active":"2024-03-26T04:11:18.250994+0000","last_peered":"2024-03-26T04:11:18.250994+0000","last_clean":"2024-03-26T04:11:18.250994+0000","last_became_active":"2024-03-26T04:10:53.561090+0000","last_became_peered":"2024-03-26T04:10:53.561090+0000","last_unstale":"2024-03-26T04:11:18.250994+0000","last_undegraded":"2024-03-26T04:11:18.250994+0000","last_fullsized":"2024-03-26T04:11:18.250994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:19:14.305912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248459+0000","last_change":"2024-03-26T04:10:57.587673+0000","last_active":"2024-03-26T04:11:18.248459+0000","last_peered":"2024-03-26T04:11:18.248459+0000","last_clean":"2024-03-26T04:11:18.248459+0000","last_became_active":"2024-03-26T04:10:57.586424+0000","last_became_peered":"2024-03-26T04:10:57.586424+0000","last_unstale":"2024-03-26T04:11:18.248459+0000","last_undegraded":"2024-03-26T04:11:18.248459+0000","last_fullsized":"2024-03-26T04:11:18.248459+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:16:07.181924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163056+0000","last_change":"2024-03-26T04:10:55.576080+0000","last_active":"2024-03-26T04:11:18.163056+0000","last_peered":"2024-03-26T04:11:18.163056+0000","last_clean":"2024-03-26T04:11:18.163056+0000","last_became_active":"2024-03-26T04:10:55.575740+0000","last_became_peered":"2024-03-26T04:10:55.575740+0000","last_unstale":"2024-03-26T04:11:18.163056+0000","last_undegraded":"2024-03-26T04:11:18.163056+0000","last_fullsized":"2024-03-26T04:11:18.163056+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:26:16.418596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251328+0000","last_change":"2024-03-26T04:10:51.541668+0000","last_active":"2024-03-26T04:11:18.251328+0000","last_peered":"2024-03-26T04:11:18.251328+0000","last_clean":"2024-03-26T04:11:18.251328+0000","last_became_active":"2024-03-26T04:10:51.541122+0000","last_became_peered":"2024-03-26T04:10:51.541122+0000","last_unstale":"2024-03-26T04:11:18.251328+0000","last_undegraded":"2024-03-26T04:11:18.251328+0000","last_fullsized":"2024-03-26T04:11:18.251328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:06:03.006022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250720+0000","last_change":"2024-03-26T04:10:53.560066+0000","last_active":"2024-03-26T04:11:18.250720+0000","last_peered":"2024-03-26T04:11:18.250720+0000","last_clean":"2024-03-26T04:11:18.250720+0000","last_became_active":"2024-03-26T04:10:53.559704+0000","last_became_peered":"2024-03-26T04:10:53.559704+0000","last_unstale":"2024-03-26T04:11:18.250720+0000","last_undegraded":"2024-03-26T04:11:18.250720+0000","last_fullsized":"2024-03-26T04:11:18.250720+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:18:40.816885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162066+0000","last_change":"2024-03-26T04:10:57.585660+0000","last_active":"2024-03-26T04:11:18.162066+0000","last_peered":"2024-03-26T04:11:18.162066+0000","last_clean":"2024-03-26T04:11:18.162066+0000","last_became_active":"2024-03-26T04:10:57.585070+0000","last_became_peered":"2024-03-26T04:10:57.585070+0000","last_unstale":"2024-03-26T04:11:18.162066+0000","last_undegraded":"2024-03-26T04:11:18.162066+0000","last_fullsized":"2024-03-26T04:11:18.162066+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:52:21.766227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163790+0000","last_change":"2024-03-26T04:10:55.575317+0000","last_active":"2024-03-26T04:11:18.163790+0000","last_peered":"2024-03-26T04:11:18.163790+0000","last_clean":"2024-03-26T04:11:18.163790+0000","last_became_active":"2024-03-26T04:10:55.574475+0000","last_became_peered":"2024-03-26T04:10:55.574475+0000","last_unstale":"2024-03-26T04:11:18.163790+0000","last_undegraded":"2024-03-26T04:11:18.163790+0000","last_fullsized":"2024-03-26T04:11:18.163790+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:50:16.780131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165937+0000","last_change":"2024-03-26T04:10:51.542609+0000","last_active":"2024-03-26T04:11:18.165937+0000","last_peered":"2024-03-26T04:11:18.165937+0000","last_clean":"2024-03-26T04:11:18.165937+0000","last_became_active":"2024-03-26T04:10:51.542330+0000","last_became_peered":"2024-03-26T04:10:51.542330+0000","last_unstale":"2024-03-26T04:11:18.165937+0000","last_undegraded":"2024-03-26T04:11:18.165937+0000","last_fullsized":"2024-03-26T04:11:18.165937+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:02:41.765004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164082+0000","last_change":"2024-03-26T04:10:53.563395+0000","last_active":"2024-03-26T04:11:18.164082+0000","last_peered":"2024-03-26T04:11:18.164082+0000","last_clean":"2024-03-26T04:11:18.164082+0000","last_became_active":"2024-03-26T04:10:53.562987+0000","last_became_peered":"2024-03-26T04:10:53.562987+0000","last_unstale":"2024-03-26T04:11:18.164082+0000","last_undegraded":"2024-03-26T04:11:18.164082+0000","last_fullsized":"2024-03-26T04:11:18.164082+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:36:10.377852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163536+0000","last_change":"2024-03-26T04:10:57.592688+0000","last_active":"2024-03-26T04:11:18.163536+0000","last_peered":"2024-03-26T04:11:18.163536+0000","last_clean":"2024-03-26T04:11:18.163536+0000","last_became_active":"2024-03-26T04:10:57.591008+0000","last_became_peered":"2024-03-26T04:10:57.591008+0000","last_unstale":"2024-03-26T04:11:18.163536+0000","last_undegraded":"2024-03-26T04:11:18.163536+0000","last_fullsized":"2024-03-26T04:11:18.163536+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:18:08.773881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163043+0000","last_change":"2024-03-26T04:10:55.575606+0000","last_active":"2024-03-26T04:11:18.163043+0000","last_peered":"2024-03-26T04:11:18.163043+0000","last_clean":"2024-03-26T04:11:18.163043+0000","last_became_active":"2024-03-26T04:10:55.574646+0000","last_became_peered":"2024-03-26T04:10:55.574646+0000","last_unstale":"2024-03-26T04:11:18.163043+0000","last_undegraded":"2024-03-26T04:11:18.163043+0000","last_fullsized":"2024-03-26T04:11:18.163043+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:27:58.908247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163092+0000","last_change":"2024-03-26T04:10:51.539728+0000","last_active":"2024-03-26T04:11:18.163092+0000","last_peered":"2024-03-26T04:11:18.163092+0000","last_clean":"2024-03-26T04:11:18.163092+0000","last_became_active":"2024-03-26T04:10:51.539369+0000","last_became_peered":"2024-03-26T04:10:51.539369+0000","last_unstale":"2024-03-26T04:11:18.163092+0000","last_undegraded":"2024-03-26T04:11:18.163092+0000","last_fullsized":"2024-03-26T04:11:18.163092+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:19:44.450425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'20","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165819+0000","last_change":"2024-03-26T04:10:53.563880+0000","last_active":"2024-03-26T04:11:18.165819+0000","last_peered":"2024-03-26T04:11:18.165819+0000","last_clean":"2024-03-26T04:11:18.165819+0000","last_became_active":"2024-03-26T04:10:53.563564+0000","last_became_peered":"2024-03-26T04:10:53.563564+0000","last_unstale":"2024-03-26T04:11:18.165819+0000","last_undegraded":"2024-03-26T04:11:18.165819+0000","last_fullsized":"2024-03-26T04:11:18.165819+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:53:53.377906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163733+0000","last_change":"2024-03-26T04:10:57.589990+0000","last_active":"2024-03-26T04:11:18.163733+0000","last_peered":"2024-03-26T04:11:18.163733+0000","last_clean":"2024-03-26T04:11:18.163733+0000","last_became_active":"2024-03-26T04:10:57.589668+0000","last_became_peered":"2024-03-26T04:10:57.589668+0000","last_unstale":"2024-03-26T04:11:18.163733+0000","last_undegraded":"2024-03-26T04:11:18.163733+0000","last_fullsized":"2024-03-26T04:11:18.163733+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:37:03.809732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165191+0000","last_change":"2024-03-26T04:10:55.572450+0000","last_active":"2024-03-26T04:11:18.165191+0000","last_peered":"2024-03-26T04:11:18.165191+0000","last_clean":"2024-03-26T04:11:18.165191+0000","last_became_active":"2024-03-26T04:10:55.571394+0000","last_became_peered":"2024-03-26T04:10:55.571394+0000","last_unstale":"2024-03-26T04:11:18.165191+0000","last_undegraded":"2024-03-26T04:11:18.165191+0000","last_fullsized":"2024-03-26T04:11:18.165191+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:02:03.086578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163671+0000","last_change":"2024-03-26T04:10:51.540563+0000","last_active":"2024-03-26T04:11:18.163671+0000","last_peered":"2024-03-26T04:11:18.163671+0000","last_clean":"2024-03-26T04:11:18.163671+0000","last_became_active":"2024-03-26T04:10:51.540250+0000","last_became_peered":"2024-03-26T04:10:51.540250+0000","last_unstale":"2024-03-26T04:11:18.163671+0000","last_undegraded":"2024-03-26T04:11:18.163671+0000","last_fullsized":"2024-03-26T04:11:18.163671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:54:51.397311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163451+0000","last_change":"2024-03-26T04:10:53.562420+0000","last_active":"2024-03-26T04:11:18.163451+0000","last_peered":"2024-03-26T04:11:18.163451+0000","last_clean":"2024-03-26T04:11:18.163451+0000","last_became_active":"2024-03-26T04:10:53.562088+0000","last_became_peered":"2024-03-26T04:10:53.562088+0000","last_unstale":"2024-03-26T04:11:18.163451+0000","last_undegraded":"2024-03-26T04:11:18.163451+0000","last_fullsized":"2024-03-26T04:11:18.163451+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:42:56.273619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250065+0000","last_change":"2024-03-26T04:10:57.588378+0000","last_active":"2024-03-26T04:11:18.250065+0000","last_peered":"2024-03-26T04:11:18.250065+0000","last_clean":"2024-03-26T04:11:18.250065+0000","last_became_active":"2024-03-26T04:10:57.587052+0000","last_became_peered":"2024-03-26T04:10:57.587052+0000","last_unstale":"2024-03-26T04:11:18.250065+0000","last_undegraded":"2024-03-26T04:11:18.250065+0000","last_fullsized":"2024-03-26T04:11:18.250065+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:03:08.339152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163340+0000","last_change":"2024-03-26T04:10:55.572854+0000","last_active":"2024-03-26T04:11:18.163340+0000","last_peered":"2024-03-26T04:11:18.163340+0000","last_clean":"2024-03-26T04:11:18.163340+0000","last_became_active":"2024-03-26T04:10:55.572555+0000","last_became_peered":"2024-03-26T04:10:55.572555+0000","last_unstale":"2024-03-26T04:11:18.163340+0000","last_undegraded":"2024-03-26T04:11:18.163340+0000","last_fullsized":"2024-03-26T04:11:18.163340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:25:52.001859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163761+0000","last_change":"2024-03-26T04:10:51.545236+0000","last_active":"2024-03-26T04:11:18.163761+0000","last_peered":"2024-03-26T04:11:18.163761+0000","last_clean":"2024-03-26T04:11:18.163761+0000","last_became_active":"2024-03-26T04:10:51.544876+0000","last_became_peered":"2024-03-26T04:10:51.544876+0000","last_unstale":"2024-03-26T04:11:18.163761+0000","last_undegraded":"2024-03-26T04:11:18.163761+0000","last_fullsized":"2024-03-26T04:11:18.163761+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:20:04.457577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163427+0000","last_change":"2024-03-26T04:10:53.561304+0000","last_active":"2024-03-26T04:11:18.163427+0000","last_peered":"2024-03-26T04:11:18.163427+0000","last_clean":"2024-03-26T04:11:18.163427+0000","last_became_active":"2024-03-26T04:10:53.561003+0000","last_became_peered":"2024-03-26T04:10:53.561003+0000","last_unstale":"2024-03-26T04:11:18.163427+0000","last_undegraded":"2024-03-26T04:11:18.163427+0000","last_fullsized":"2024-03-26T04:11:18.163427+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:33:51.344073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249771+0000","last_change":"2024-03-26T04:10:57.589417+0000","last_active":"2024-03-26T04:11:18.249771+0000","last_peered":"2024-03-26T04:11:18.249771+0000","last_clean":"2024-03-26T04:11:18.249771+0000","last_became_active":"2024-03-26T04:10:57.588537+0000","last_became_peered":"2024-03-26T04:10:57.588537+0000","last_unstale":"2024-03-26T04:11:18.249771+0000","last_undegraded":"2024-03-26T04:11:18.249771+0000","last_fullsized":"2024-03-26T04:11:18.249771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:42:01.264561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:21.588306+0000","last_change":"2024-03-26T04:10:55.572583+0000","last_active":"2024-03-26T04:11:21.588306+0000","last_peered":"2024-03-26T04:11:21.588306+0000","last_clean":"2024-03-26T04:11:21.588306+0000","last_became_active":"2024-03-26T04:10:55.571273+0000","last_became_peered":"2024-03-26T04:10:55.571273+0000","last_unstale":"2024-03-26T04:11:21.588306+0000","last_undegraded":"2024-03-26T04:11:21.588306+0000","last_fullsized":"2024-03-26T04:11:21.588306+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:51:35.178605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248646+0000","last_change":"2024-03-26T04:10:51.542935+0000","last_active":"2024-03-26T04:11:18.248646+0000","last_peered":"2024-03-26T04:11:18.248646+0000","last_clean":"2024-03-26T04:11:18.248646+0000","last_became_active":"2024-03-26T04:10:51.541850+0000","last_became_peered":"2024-03-26T04:10:51.541850+0000","last_unstale":"2024-03-26T04:11:18.248646+0000","last_undegraded":"2024-03-26T04:11:18.248646+0000","last_fullsized":"2024-03-26T04:11:18.248646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:29:40.572990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248140+0000","last_change":"2024-03-26T04:10:53.561899+0000","last_active":"2024-03-26T04:11:18.248140+0000","last_peered":"2024-03-26T04:11:18.248140+0000","last_clean":"2024-03-26T04:11:18.248140+0000","last_became_active":"2024-03-26T04:10:53.561177+0000","last_became_peered":"2024-03-26T04:10:53.561177+0000","last_unstale":"2024-03-26T04:11:18.248140+0000","last_undegraded":"2024-03-26T04:11:18.248140+0000","last_fullsized":"2024-03-26T04:11:18.248140+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:17:38.210701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164473+0000","last_change":"2024-03-26T04:10:57.592801+0000","last_active":"2024-03-26T04:11:18.164473+0000","last_peered":"2024-03-26T04:11:18.164473+0000","last_clean":"2024-03-26T04:11:18.164473+0000","last_became_active":"2024-03-26T04:10:57.591597+0000","last_became_peered":"2024-03-26T04:10:57.591597+0000","last_unstale":"2024-03-26T04:11:18.164473+0000","last_undegraded":"2024-03-26T04:11:18.164473+0000","last_fullsized":"2024-03-26T04:11:18.164473+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:34:50.321529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250159+0000","last_change":"2024-03-26T04:10:55.570574+0000","last_active":"2024-03-26T04:11:18.250159+0000","last_peered":"2024-03-26T04:11:18.250159+0000","last_clean":"2024-03-26T04:11:18.250159+0000","last_became_active":"2024-03-26T04:10:55.570234+0000","last_became_peered":"2024-03-26T04:10:55.570234+0000","last_unstale":"2024-03-26T04:11:18.250159+0000","last_undegraded":"2024-03-26T04:11:18.250159+0000","last_fullsized":"2024-03-26T04:11:18.250159+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:18:44.535190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162074+0000","last_change":"2024-03-26T04:10:51.543502+0000","last_active":"2024-03-26T04:11:18.162074+0000","last_peered":"2024-03-26T04:11:18.162074+0000","last_clean":"2024-03-26T04:11:18.162074+0000","last_became_active":"2024-03-26T04:10:51.542690+0000","last_became_peered":"2024-03-26T04:10:51.542690+0000","last_unstale":"2024-03-26T04:11:18.162074+0000","last_undegraded":"2024-03-26T04:11:18.162074+0000","last_fullsized":"2024-03-26T04:11:18.162074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:27:12.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'39","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248648+0000","last_change":"2024-03-26T04:10:53.559877+0000","last_active":"2024-03-26T04:11:18.248648+0000","last_peered":"2024-03-26T04:11:18.248648+0000","last_clean":"2024-03-26T04:11:18.248648+0000","last_became_active":"2024-03-26T04:10:53.559481+0000","last_became_peered":"2024-03-26T04:10:53.559481+0000","last_unstale":"2024-03-26T04:11:18.248648+0000","last_undegraded":"2024-03-26T04:11:18.248648+0000","last_fullsized":"2024-03-26T04:11:18.248648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:54:01.903188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163214+0000","last_change":"2024-03-26T04:10:57.592925+0000","last_active":"2024-03-26T04:11:18.163214+0000","last_peered":"2024-03-26T04:11:18.163214+0000","last_clean":"2024-03-26T04:11:18.163214+0000","last_became_active":"2024-03-26T04:10:57.591484+0000","last_became_peered":"2024-03-26T04:10:57.591484+0000","last_unstale":"2024-03-26T04:11:18.163214+0000","last_undegraded":"2024-03-26T04:11:18.163214+0000","last_fullsized":"2024-03-26T04:11:18.163214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:18.678133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248340+0000","last_change":"2024-03-26T04:10:55.570968+0000","last_active":"2024-03-26T04:11:18.248340+0000","last_peered":"2024-03-26T04:11:18.248340+0000","last_clean":"2024-03-26T04:11:18.248340+0000","last_became_active":"2024-03-26T04:10:55.570332+0000","last_became_peered":"2024-03-26T04:10:55.570332+0000","last_unstale":"2024-03-26T04:11:18.248340+0000","last_undegraded":"2024-03-26T04:11:18.248340+0000","last_fullsized":"2024-03-26T04:11:18.248340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:07:43.244996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165698+0000","last_change":"2024-03-26T04:10:51.545345+0000","last_active":"2024-03-26T04:11:18.165698+0000","last_peered":"2024-03-26T04:11:18.165698+0000","last_clean":"2024-03-26T04:11:18.165698+0000","last_became_active":"2024-03-26T04:10:51.545000+0000","last_became_peered":"2024-03-26T04:10:51.545000+0000","last_unstale":"2024-03-26T04:11:18.165698+0000","last_undegraded":"2024-03-26T04:11:18.165698+0000","last_fullsized":"2024-03-26T04:11:18.165698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:16:21.191400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'22","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165791+0000","last_change":"2024-03-26T04:10:53.562655+0000","last_active":"2024-03-26T04:11:18.165791+0000","last_peered":"2024-03-26T04:11:18.165791+0000","last_clean":"2024-03-26T04:11:18.165791+0000","last_became_active":"2024-03-26T04:10:53.561153+0000","last_became_peered":"2024-03-26T04:10:53.561153+0000","last_unstale":"2024-03-26T04:11:18.165791+0000","last_undegraded":"2024-03-26T04:11:18.165791+0000","last_fullsized":"2024-03-26T04:11:18.165791+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:45:11.285844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163148+0000","last_change":"2024-03-26T04:10:57.589809+0000","last_active":"2024-03-26T04:11:18.163148+0000","last_peered":"2024-03-26T04:11:18.163148+0000","last_clean":"2024-03-26T04:11:18.163148+0000","last_became_active":"2024-03-26T04:10:57.589504+0000","last_became_peered":"2024-03-26T04:10:57.589504+0000","last_unstale":"2024-03-26T04:11:18.163148+0000","last_undegraded":"2024-03-26T04:11:18.163148+0000","last_fullsized":"2024-03-26T04:11:18.163148+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:07:07.029474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248564+0000","last_change":"2024-03-26T04:10:55.570631+0000","last_active":"2024-03-26T04:11:18.248564+0000","last_peered":"2024-03-26T04:11:18.248564+0000","last_clean":"2024-03-26T04:11:18.248564+0000","last_became_active":"2024-03-26T04:10:55.570198+0000","last_became_peered":"2024-03-26T04:10:55.570198+0000","last_unstale":"2024-03-26T04:11:18.248564+0000","last_undegraded":"2024-03-26T04:11:18.248564+0000","last_fullsized":"2024-03-26T04:11:18.248564+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:16:43.055656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165491+0000","last_change":"2024-03-26T04:10:51.542723+0000","last_active":"2024-03-26T04:11:18.165491+0000","last_peered":"2024-03-26T04:11:18.165491+0000","last_clean":"2024-03-26T04:11:18.165491+0000","last_became_active":"2024-03-26T04:10:51.542430+0000","last_became_peered":"2024-03-26T04:10:51.542430+0000","last_unstale":"2024-03-26T04:11:18.165491+0000","last_undegraded":"2024-03-26T04:11:18.165491+0000","last_fullsized":"2024-03-26T04:11:18.165491+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:34:46.065266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251159+0000","last_change":"2024-03-26T04:10:53.561834+0000","last_active":"2024-03-26T04:11:18.251159+0000","last_peered":"2024-03-26T04:11:18.251159+0000","last_clean":"2024-03-26T04:11:18.251159+0000","last_became_active":"2024-03-26T04:10:53.561132+0000","last_became_peered":"2024-03-26T04:10:53.561132+0000","last_unstale":"2024-03-26T04:11:18.251159+0000","last_undegraded":"2024-03-26T04:11:18.251159+0000","last_fullsized":"2024-03-26T04:11:18.251159+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:29:31.297404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162357+0000","last_change":"2024-03-26T04:10:57.585780+0000","last_active":"2024-03-26T04:11:18.162357+0000","last_peered":"2024-03-26T04:11:18.162357+0000","last_clean":"2024-03-26T04:11:18.162357+0000","last_became_active":"2024-03-26T04:10:57.585489+0000","last_became_peered":"2024-03-26T04:10:57.585489+0000","last_unstale":"2024-03-26T04:11:18.162357+0000","last_undegraded":"2024-03-26T04:11:18.162357+0000","last_fullsized":"2024-03-26T04:11:18.162357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:37:09.101121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164366+0000","last_change":"2024-03-26T04:10:55.575328+0000","last_active":"2024-03-26T04:11:18.164366+0000","last_peered":"2024-03-26T04:11:18.164366+0000","last_clean":"2024-03-26T04:11:18.164366+0000","last_became_active":"2024-03-26T04:10:55.574415+0000","last_became_peered":"2024-03-26T04:10:55.574415+0000","last_unstale":"2024-03-26T04:11:18.164366+0000","last_undegraded":"2024-03-26T04:11:18.164366+0000","last_fullsized":"2024-03-26T04:11:18.164366+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:06.530199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165773+0000","last_change":"2024-03-26T04:10:51.541500+0000","last_active":"2024-03-26T04:11:18.165773+0000","last_peered":"2024-03-26T04:11:18.165773+0000","last_clean":"2024-03-26T04:11:18.165773+0000","last_became_active":"2024-03-26T04:10:51.541204+0000","last_became_peered":"2024-03-26T04:10:51.541204+0000","last_unstale":"2024-03-26T04:11:18.165773+0000","last_undegraded":"2024-03-26T04:11:18.165773+0000","last_fullsized":"2024-03-26T04:11:18.165773+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:15:24.926305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164268+0000","last_change":"2024-03-26T04:10:53.561793+0000","last_active":"2024-03-26T04:11:18.164268+0000","last_peered":"2024-03-26T04:11:18.164268+0000","last_clean":"2024-03-26T04:11:18.164268+0000","last_became_active":"2024-03-26T04:10:53.561017+0000","last_became_peered":"2024-03-26T04:10:53.561017+0000","last_unstale":"2024-03-26T04:11:18.164268+0000","last_undegraded":"2024-03-26T04:11:18.164268+0000","last_fullsized":"2024-03-26T04:11:18.164268+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:31:24.536019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162569+0000","last_change":"2024-03-26T04:10:57.585277+0000","last_active":"2024-03-26T04:11:18.162569+0000","last_peered":"2024-03-26T04:11:18.162569+0000","last_clean":"2024-03-26T04:11:18.162569+0000","last_became_active":"2024-03-26T04:10:57.584678+0000","last_became_peered":"2024-03-26T04:10:57.584678+0000","last_unstale":"2024-03-26T04:11:18.162569+0000","last_undegraded":"2024-03-26T04:11:18.162569+0000","last_fullsized":"2024-03-26T04:11:18.162569+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:44:26.960546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249009+0000","last_change":"2024-03-26T04:10:55.569107+0000","last_active":"2024-03-26T04:11:18.249009+0000","last_peered":"2024-03-26T04:11:18.249009+0000","last_clean":"2024-03-26T04:11:18.249009+0000","last_became_active":"2024-03-26T04:10:55.568739+0000","last_became_peered":"2024-03-26T04:10:55.568739+0000","last_unstale":"2024-03-26T04:11:18.249009+0000","last_undegraded":"2024-03-26T04:11:18.249009+0000","last_fullsized":"2024-03-26T04:11:18.249009+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:42:27.416428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164173+0000","last_change":"2024-03-26T04:10:51.545202+0000","last_active":"2024-03-26T04:11:18.164173+0000","last_peered":"2024-03-26T04:11:18.164173+0000","last_clean":"2024-03-26T04:11:18.164173+0000","last_became_active":"2024-03-26T04:10:51.544915+0000","last_became_peered":"2024-03-26T04:10:51.544915+0000","last_unstale":"2024-03-26T04:11:18.164173+0000","last_undegraded":"2024-03-26T04:11:18.164173+0000","last_fullsized":"2024-03-26T04:11:18.164173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:18:13.441979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165526+0000","last_change":"2024-03-26T04:10:53.562613+0000","last_active":"2024-03-26T04:11:18.165526+0000","last_peered":"2024-03-26T04:11:18.165526+0000","last_clean":"2024-03-26T04:11:18.165526+0000","last_became_active":"2024-03-26T04:10:53.562311+0000","last_became_peered":"2024-03-26T04:10:53.562311+0000","last_unstale":"2024-03-26T04:11:18.165526+0000","last_undegraded":"2024-03-26T04:11:18.165526+0000","last_fullsized":"2024-03-26T04:11:18.165526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:34:23.272447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249672+0000","last_change":"2024-03-26T04:10:57.587709+0000","last_active":"2024-03-26T04:11:18.249672+0000","last_peered":"2024-03-26T04:11:18.249672+0000","last_clean":"2024-03-26T04:11:18.249672+0000","last_became_active":"2024-03-26T04:10:57.587117+0000","last_became_peered":"2024-03-26T04:10:57.587117+0000","last_unstale":"2024-03-26T04:11:18.249672+0000","last_undegraded":"2024-03-26T04:11:18.249672+0000","last_fullsized":"2024-03-26T04:11:18.249672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:03:34.379361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:21.588337+0000","last_change":"2024-03-26T04:10:55.573195+0000","last_active":"2024-03-26T04:11:21.588337+0000","last_peered":"2024-03-26T04:11:21.588337+0000","last_clean":"2024-03-26T04:11:21.588337+0000","last_became_active":"2024-03-26T04:10:55.571369+0000","last_became_peered":"2024-03-26T04:10:55.571369+0000","last_unstale":"2024-03-26T04:11:21.588337+0000","last_undegraded":"2024-03-26T04:11:21.588337+0000","last_fullsized":"2024-03-26T04:11:21.588337+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:32:10.267637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163202+0000","last_change":"2024-03-26T04:10:51.541539+0000","last_active":"2024-03-26T04:11:18.163202+0000","last_peered":"2024-03-26T04:11:18.163202+0000","last_clean":"2024-03-26T04:11:18.163202+0000","last_became_active":"2024-03-26T04:10:51.541247+0000","last_became_peered":"2024-03-26T04:10:51.541247+0000","last_unstale":"2024-03-26T04:11:18.163202+0000","last_undegraded":"2024-03-26T04:11:18.163202+0000","last_fullsized":"2024-03-26T04:11:18.163202+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:13:48.617563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249396+0000","last_change":"2024-03-26T04:10:53.558121+0000","last_active":"2024-03-26T04:11:18.249396+0000","last_peered":"2024-03-26T04:11:18.249396+0000","last_clean":"2024-03-26T04:11:18.249396+0000","last_became_active":"2024-03-26T04:10:53.557621+0000","last_became_peered":"2024-03-26T04:10:53.557621+0000","last_unstale":"2024-03-26T04:11:18.249396+0000","last_undegraded":"2024-03-26T04:11:18.249396+0000","last_fullsized":"2024-03-26T04:11:18.249396+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:28:39.538916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163924+0000","last_change":"2024-03-26T04:10:57.593608+0000","last_active":"2024-03-26T04:11:18.163924+0000","last_peered":"2024-03-26T04:11:18.163924+0000","last_clean":"2024-03-26T04:11:18.163924+0000","last_became_active":"2024-03-26T04:10:57.593262+0000","last_became_peered":"2024-03-26T04:10:57.593262+0000","last_unstale":"2024-03-26T04:11:18.163924+0000","last_undegraded":"2024-03-26T04:11:18.163924+0000","last_fullsized":"2024-03-26T04:11:18.163924+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:55:40.547438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:21.588366+0000","last_change":"2024-03-26T04:10:55.571420+0000","last_active":"2024-03-26T04:11:21.588366+0000","last_peered":"2024-03-26T04:11:21.588366+0000","last_clean":"2024-03-26T04:11:21.588366+0000","last_became_active":"2024-03-26T04:10:55.571113+0000","last_became_peered":"2024-03-26T04:10:55.571113+0000","last_unstale":"2024-03-26T04:11:21.588366+0000","last_undegraded":"2024-03-26T04:11:21.588366+0000","last_fullsized":"2024-03-26T04:11:21.588366+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:52:24.795919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162741+0000","last_change":"2024-03-26T04:10:51.539172+0000","last_active":"2024-03-26T04:11:18.162741+0000","last_peered":"2024-03-26T04:11:18.162741+0000","last_clean":"2024-03-26T04:11:18.162741+0000","last_became_active":"2024-03-26T04:10:51.538874+0000","last_became_peered":"2024-03-26T04:10:51.538874+0000","last_unstale":"2024-03-26T04:11:18.162741+0000","last_undegraded":"2024-03-26T04:11:18.162741+0000","last_fullsized":"2024-03-26T04:11:18.162741+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:34:25.716285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164312+0000","last_change":"2024-03-26T04:10:53.564494+0000","last_active":"2024-03-26T04:11:18.164312+0000","last_peered":"2024-03-26T04:11:18.164312+0000","last_clean":"2024-03-26T04:11:18.164312+0000","last_became_active":"2024-03-26T04:10:53.563661+0000","last_became_peered":"2024-03-26T04:10:53.563661+0000","last_unstale":"2024-03-26T04:11:18.164312+0000","last_undegraded":"2024-03-26T04:11:18.164312+0000","last_fullsized":"2024-03-26T04:11:18.164312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:14:49.234160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164457+0000","last_change":"2024-03-26T04:10:57.593462+0000","last_active":"2024-03-26T04:11:18.164457+0000","last_peered":"2024-03-26T04:11:18.164457+0000","last_clean":"2024-03-26T04:11:18.164457+0000","last_became_active":"2024-03-26T04:10:57.593153+0000","last_became_peered":"2024-03-26T04:10:57.593153+0000","last_unstale":"2024-03-26T04:11:18.164457+0000","last_undegraded":"2024-03-26T04:11:18.164457+0000","last_fullsized":"2024-03-26T04:11:18.164457+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:27:23.253305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251143+0000","last_change":"2024-03-26T04:10:55.572211+0000","last_active":"2024-03-26T04:11:18.251143+0000","last_peered":"2024-03-26T04:11:18.251143+0000","last_clean":"2024-03-26T04:11:18.251143+0000","last_became_active":"2024-03-26T04:10:55.571085+0000","last_became_peered":"2024-03-26T04:10:55.571085+0000","last_unstale":"2024-03-26T04:11:18.251143+0000","last_undegraded":"2024-03-26T04:11:18.251143+0000","last_fullsized":"2024-03-26T04:11:18.251143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:31:48.287587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163414+0000","last_change":"2024-03-26T04:10:51.545279+0000","last_active":"2024-03-26T04:11:18.163414+0000","last_peered":"2024-03-26T04:11:18.163414+0000","last_clean":"2024-03-26T04:11:18.163414+0000","last_became_active":"2024-03-26T04:10:51.544891+0000","last_became_peered":"2024-03-26T04:10:51.544891+0000","last_unstale":"2024-03-26T04:11:18.163414+0000","last_undegraded":"2024-03-26T04:11:18.163414+0000","last_fullsized":"2024-03-26T04:11:18.163414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:20:04.302284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162816+0000","last_change":"2024-03-26T04:10:53.565015+0000","last_active":"2024-03-26T04:11:18.162816+0000","last_peered":"2024-03-26T04:11:18.162816+0000","last_clean":"2024-03-26T04:11:18.162816+0000","last_became_active":"2024-03-26T04:10:53.564631+0000","last_became_peered":"2024-03-26T04:10:53.564631+0000","last_unstale":"2024-03-26T04:11:18.162816+0000","last_undegraded":"2024-03-26T04:11:18.162816+0000","last_fullsized":"2024-03-26T04:11:18.162816+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:31:27.784195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249667+0000","last_change":"2024-03-26T04:10:57.587248+0000","last_active":"2024-03-26T04:11:18.249667+0000","last_peered":"2024-03-26T04:11:18.249667+0000","last_clean":"2024-03-26T04:11:18.249667+0000","last_became_active":"2024-03-26T04:10:57.586726+0000","last_became_peered":"2024-03-26T04:10:57.586726+0000","last_unstale":"2024-03-26T04:11:18.249667+0000","last_undegraded":"2024-03-26T04:11:18.249667+0000","last_fullsized":"2024-03-26T04:11:18.249667+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:22:31.701143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250183+0000","last_change":"2024-03-26T04:10:55.570650+0000","last_active":"2024-03-26T04:11:18.250183+0000","last_peered":"2024-03-26T04:11:18.250183+0000","last_clean":"2024-03-26T04:11:18.250183+0000","last_became_active":"2024-03-26T04:10:55.569000+0000","last_became_peered":"2024-03-26T04:10:55.569000+0000","last_unstale":"2024-03-26T04:11:18.250183+0000","last_undegraded":"2024-03-26T04:11:18.250183+0000","last_fullsized":"2024-03-26T04:11:18.250183+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:31:34.623827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162207+0000","last_change":"2024-03-26T04:10:51.542655+0000","last_active":"2024-03-26T04:11:18.162207+0000","last_peered":"2024-03-26T04:11:18.162207+0000","last_clean":"2024-03-26T04:11:18.162207+0000","last_became_active":"2024-03-26T04:10:51.542153+0000","last_became_peered":"2024-03-26T04:10:51.542153+0000","last_unstale":"2024-03-26T04:11:18.162207+0000","last_undegraded":"2024-03-26T04:11:18.162207+0000","last_fullsized":"2024-03-26T04:11:18.162207+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:08:20.158401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162801+0000","last_change":"2024-03-26T04:10:53.561470+0000","last_active":"2024-03-26T04:11:18.162801+0000","last_peered":"2024-03-26T04:11:18.162801+0000","last_clean":"2024-03-26T04:11:18.162801+0000","last_became_active":"2024-03-26T04:10:53.560843+0000","last_became_peered":"2024-03-26T04:10:53.560843+0000","last_unstale":"2024-03-26T04:11:18.162801+0000","last_undegraded":"2024-03-26T04:11:18.162801+0000","last_fullsized":"2024-03-26T04:11:18.162801+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:30:17.446361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162473+0000","last_change":"2024-03-26T04:10:57.585543+0000","last_active":"2024-03-26T04:11:18.162473+0000","last_peered":"2024-03-26T04:11:18.162473+0000","last_clean":"2024-03-26T04:11:18.162473+0000","last_became_active":"2024-03-26T04:10:57.584803+0000","last_became_peered":"2024-03-26T04:10:57.584803+0000","last_unstale":"2024-03-26T04:11:18.162473+0000","last_undegraded":"2024-03-26T04:11:18.162473+0000","last_fullsized":"2024-03-26T04:11:18.162473+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:05:14.911209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164790+0000","last_change":"2024-03-26T04:10:51.541599+0000","last_active":"2024-03-26T04:11:18.164790+0000","last_peered":"2024-03-26T04:11:18.164790+0000","last_clean":"2024-03-26T04:11:18.164790+0000","last_became_active":"2024-03-26T04:10:51.541262+0000","last_became_peered":"2024-03-26T04:10:51.541262+0000","last_unstale":"2024-03-26T04:11:18.164790+0000","last_undegraded":"2024-03-26T04:11:18.164790+0000","last_fullsized":"2024-03-26T04:11:18.164790+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:57:13.317922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163906+0000","last_change":"2024-03-26T04:10:53.562649+0000","last_active":"2024-03-26T04:11:18.163906+0000","last_peered":"2024-03-26T04:11:18.163906+0000","last_clean":"2024-03-26T04:11:18.163906+0000","last_became_active":"2024-03-26T04:10:53.562347+0000","last_became_peered":"2024-03-26T04:10:53.562347+0000","last_unstale":"2024-03-26T04:11:18.163906+0000","last_undegraded":"2024-03-26T04:11:18.163906+0000","last_fullsized":"2024-03-26T04:11:18.163906+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:10:08.481378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:21.588382+0000","last_change":"2024-03-26T04:10:55.570854+0000","last_active":"2024-03-26T04:11:21.588382+0000","last_peered":"2024-03-26T04:11:21.588382+0000","last_clean":"2024-03-26T04:11:21.588382+0000","last_became_active":"2024-03-26T04:10:55.570398+0000","last_became_peered":"2024-03-26T04:10:55.570398+0000","last_unstale":"2024-03-26T04:11:21.588382+0000","last_undegraded":"2024-03-26T04:11:21.588382+0000","last_fullsized":"2024-03-26T04:11:21.588382+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:19:35.228456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249246+0000","last_change":"2024-03-26T04:10:57.587403+0000","last_active":"2024-03-26T04:11:18.249246+0000","last_peered":"2024-03-26T04:11:18.249246+0000","last_clean":"2024-03-26T04:11:18.249246+0000","last_became_active":"2024-03-26T04:10:57.586769+0000","last_became_peered":"2024-03-26T04:10:57.586769+0000","last_unstale":"2024-03-26T04:11:18.249246+0000","last_undegraded":"2024-03-26T04:11:18.249246+0000","last_fullsized":"2024-03-26T04:11:18.249246+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:24:57.514820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250714+0000","last_change":"2024-03-26T04:10:51.541556+0000","last_active":"2024-03-26T04:11:18.250714+0000","last_peered":"2024-03-26T04:11:18.250714+0000","last_clean":"2024-03-26T04:11:18.250714+0000","last_became_active":"2024-03-26T04:10:51.540912+0000","last_became_peered":"2024-03-26T04:10:51.540912+0000","last_unstale":"2024-03-26T04:11:18.250714+0000","last_undegraded":"2024-03-26T04:11:18.250714+0000","last_fullsized":"2024-03-26T04:11:18.250714+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:50:09.093505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:21.588195+0000","last_change":"2024-03-26T04:10:53.561394+0000","last_active":"2024-03-26T04:11:21.588195+0000","last_peered":"2024-03-26T04:11:21.588195+0000","last_clean":"2024-03-26T04:11:21.588195+0000","last_became_active":"2024-03-26T04:10:53.560920+0000","last_became_peered":"2024-03-26T04:10:53.560920+0000","last_unstale":"2024-03-26T04:11:21.588195+0000","last_undegraded":"2024-03-26T04:11:21.588195+0000","last_fullsized":"2024-03-26T04:11:21.588195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:32:30.124529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162462+0000","last_change":"2024-03-26T04:10:55.570775+0000","last_active":"2024-03-26T04:11:18.162462+0000","last_peered":"2024-03-26T04:11:18.162462+0000","last_clean":"2024-03-26T04:11:18.162462+0000","last_became_active":"2024-03-26T04:10:55.570451+0000","last_became_peered":"2024-03-26T04:10:55.570451+0000","last_unstale":"2024-03-26T04:11:18.162462+0000","last_undegraded":"2024-03-26T04:11:18.162462+0000","last_fullsized":"2024-03-26T04:11:18.162462+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:50.696483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249498+0000","last_change":"2024-03-26T04:10:57.589035+0000","last_active":"2024-03-26T04:11:18.249498+0000","last_peered":"2024-03-26T04:11:18.249498+0000","last_clean":"2024-03-26T04:11:18.249498+0000","last_became_active":"2024-03-26T04:10:57.587980+0000","last_became_peered":"2024-03-26T04:10:57.587980+0000","last_unstale":"2024-03-26T04:11:18.249498+0000","last_undegraded":"2024-03-26T04:11:18.249498+0000","last_fullsized":"2024-03-26T04:11:18.249498+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:59.983185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163672+0000","last_change":"2024-03-26T04:10:51.545447+0000","last_active":"2024-03-26T04:11:18.163672+0000","last_peered":"2024-03-26T04:11:18.163672+0000","last_clean":"2024-03-26T04:11:18.163672+0000","last_became_active":"2024-03-26T04:10:51.545063+0000","last_became_peered":"2024-03-26T04:10:51.545063+0000","last_unstale":"2024-03-26T04:11:18.163672+0000","last_undegraded":"2024-03-26T04:11:18.163672+0000","last_fullsized":"2024-03-26T04:11:18.163672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:00:54.996095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'25","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164739+0000","last_change":"2024-03-26T04:10:53.561342+0000","last_active":"2024-03-26T04:11:18.164739+0000","last_peered":"2024-03-26T04:11:18.164739+0000","last_clean":"2024-03-26T04:11:18.164739+0000","last_became_active":"2024-03-26T04:10:53.560905+0000","last_became_peered":"2024-03-26T04:10:53.560905+0000","last_unstale":"2024-03-26T04:11:18.164739+0000","last_undegraded":"2024-03-26T04:11:18.164739+0000","last_fullsized":"2024-03-26T04:11:18.164739+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:16:46.946886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249761+0000","last_change":"2024-03-26T04:10:55.569128+0000","last_active":"2024-03-26T04:11:18.249761+0000","last_peered":"2024-03-26T04:11:18.249761+0000","last_clean":"2024-03-26T04:11:18.249761+0000","last_became_active":"2024-03-26T04:10:55.568630+0000","last_became_peered":"2024-03-26T04:10:55.568630+0000","last_unstale":"2024-03-26T04:11:18.249761+0000","last_undegraded":"2024-03-26T04:11:18.249761+0000","last_fullsized":"2024-03-26T04:11:18.249761+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:27:56.020961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251344+0000","last_change":"2024-03-26T04:10:57.589491+0000","last_active":"2024-03-26T04:11:18.251344+0000","last_peered":"2024-03-26T04:11:18.251344+0000","last_clean":"2024-03-26T04:11:18.251344+0000","last_became_active":"2024-03-26T04:10:57.589153+0000","last_became_peered":"2024-03-26T04:10:57.589153+0000","last_unstale":"2024-03-26T04:11:18.251344+0000","last_undegraded":"2024-03-26T04:11:18.251344+0000","last_fullsized":"2024-03-26T04:11:18.251344+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:46:32.206524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.161845+0000","last_change":"2024-03-26T04:10:53.561369+0000","last_active":"2024-03-26T04:11:18.161845+0000","last_peered":"2024-03-26T04:11:18.161845+0000","last_clean":"2024-03-26T04:11:18.161845+0000","last_became_active":"2024-03-26T04:10:53.560904+0000","last_became_peered":"2024-03-26T04:10:53.560904+0000","last_unstale":"2024-03-26T04:11:18.161845+0000","last_undegraded":"2024-03-26T04:11:18.161845+0000","last_fullsized":"2024-03-26T04:11:18.161845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:31:52.910869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248567+0000","last_change":"2024-03-26T04:10:51.541472+0000","last_active":"2024-03-26T04:11:18.248567+0000","last_peered":"2024-03-26T04:11:18.248567+0000","last_clean":"2024-03-26T04:11:18.248567+0000","last_became_active":"2024-03-26T04:10:51.540938+0000","last_became_peered":"2024-03-26T04:10:51.540938+0000","last_unstale":"2024-03-26T04:11:18.248567+0000","last_undegraded":"2024-03-26T04:11:18.248567+0000","last_fullsized":"2024-03-26T04:11:18.248567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:46:19.914215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251073+0000","last_change":"2024-03-26T04:10:55.570686+0000","last_active":"2024-03-26T04:11:18.251073+0000","last_peered":"2024-03-26T04:11:18.251073+0000","last_clean":"2024-03-26T04:11:18.251073+0000","last_became_active":"2024-03-26T04:10:55.570345+0000","last_became_peered":"2024-03-26T04:10:55.570345+0000","last_unstale":"2024-03-26T04:11:18.251073+0000","last_undegraded":"2024-03-26T04:11:18.251073+0000","last_fullsized":"2024-03-26T04:11:18.251073+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:22:53.826613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162461+0000","last_change":"2024-03-26T04:10:57.585569+0000","last_active":"2024-03-26T04:11:18.162461+0000","last_peered":"2024-03-26T04:11:18.162461+0000","last_clean":"2024-03-26T04:11:18.162461+0000","last_became_active":"2024-03-26T04:10:57.585280+0000","last_became_peered":"2024-03-26T04:10:57.585280+0000","last_unstale":"2024-03-26T04:11:18.162461+0000","last_undegraded":"2024-03-26T04:11:18.162461+0000","last_fullsized":"2024-03-26T04:11:18.162461+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:20:33.540628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249216+0000","last_change":"2024-03-26T04:10:53.559212+0000","last_active":"2024-03-26T04:11:18.249216+0000","last_peered":"2024-03-26T04:11:18.249216+0000","last_clean":"2024-03-26T04:11:18.249216+0000","last_became_active":"2024-03-26T04:10:53.558856+0000","last_became_peered":"2024-03-26T04:10:53.558856+0000","last_unstale":"2024-03-26T04:11:18.249216+0000","last_undegraded":"2024-03-26T04:11:18.249216+0000","last_fullsized":"2024-03-26T04:11:18.249216+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:14:31.138260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164649+0000","last_change":"2024-03-26T04:10:51.545529+0000","last_active":"2024-03-26T04:11:18.164649+0000","last_peered":"2024-03-26T04:11:18.164649+0000","last_clean":"2024-03-26T04:11:18.164649+0000","last_became_active":"2024-03-26T04:10:51.545074+0000","last_became_peered":"2024-03-26T04:10:51.545074+0000","last_unstale":"2024-03-26T04:11:18.164649+0000","last_undegraded":"2024-03-26T04:11:18.164649+0000","last_fullsized":"2024-03-26T04:11:18.164649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:12:14.151922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:20.259642+0000","last_change":"2024-03-26T04:10:37.787106+0000","last_active":"2024-03-26T04:11:20.259642+0000","last_peered":"2024-03-26T04:11:20.259642+0000","last_clean":"2024-03-26T04:11:20.259642+0000","last_became_active":"2024-03-26T04:10:37.786056+0000","last_became_peered":"2024-03-26T04:10:37.786056+0000","last_unstale":"2024-03-26T04:11:20.259642+0000","last_undegraded":"2024-03-26T04:11:20.259642+0000","last_fullsized":"2024-03-26T04:11:20.259642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:36.777647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:36.777647+0000","last_clean_scrub_stamp":"2024-03-26T04:10:36.777647+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:27:47.025092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164459+0000","last_change":"2024-03-26T04:10:55.576476+0000","last_active":"2024-03-26T04:11:18.164459+0000","last_peered":"2024-03-26T04:11:18.164459+0000","last_clean":"2024-03-26T04:11:18.164459+0000","last_became_active":"2024-03-26T04:10:55.576206+0000","last_became_peered":"2024-03-26T04:10:55.576206+0000","last_unstale":"2024-03-26T04:11:18.164459+0000","last_undegraded":"2024-03-26T04:11:18.164459+0000","last_fullsized":"2024-03-26T04:11:18.164459+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:17:16.303231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165298+0000","last_change":"2024-03-26T04:10:57.591736+0000","last_active":"2024-03-26T04:11:18.165298+0000","last_peered":"2024-03-26T04:11:18.165298+0000","last_clean":"2024-03-26T04:11:18.165298+0000","last_became_active":"2024-03-26T04:10:57.590324+0000","last_became_peered":"2024-03-26T04:10:57.590324+0000","last_unstale":"2024-03-26T04:11:18.165298+0000","last_undegraded":"2024-03-26T04:11:18.165298+0000","last_fullsized":"2024-03-26T04:11:18.165298+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:56:35.234213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248804+0000","last_change":"2024-03-26T04:10:53.558715+0000","last_active":"2024-03-26T04:11:18.248804+0000","last_peered":"2024-03-26T04:11:18.248804+0000","last_clean":"2024-03-26T04:11:18.248804+0000","last_became_active":"2024-03-26T04:10:53.557969+0000","last_became_peered":"2024-03-26T04:10:53.557969+0000","last_unstale":"2024-03-26T04:11:18.248804+0000","last_undegraded":"2024-03-26T04:11:18.248804+0000","last_fullsized":"2024-03-26T04:11:18.248804+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:07:26.089390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162751+0000","last_change":"2024-03-26T04:10:51.545819+0000","last_active":"2024-03-26T04:11:18.162751+0000","last_peered":"2024-03-26T04:11:18.162751+0000","last_clean":"2024-03-26T04:11:18.162751+0000","last_became_active":"2024-03-26T04:10:51.545569+0000","last_became_peered":"2024-03-26T04:10:51.545569+0000","last_unstale":"2024-03-26T04:11:18.162751+0000","last_undegraded":"2024-03-26T04:11:18.162751+0000","last_fullsized":"2024-03-26T04:11:18.162751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:51:29.892523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163302+0000","last_change":"2024-03-26T04:10:55.571433+0000","last_active":"2024-03-26T04:11:18.163302+0000","last_peered":"2024-03-26T04:11:18.163302+0000","last_clean":"2024-03-26T04:11:18.163302+0000","last_became_active":"2024-03-26T04:10:55.570925+0000","last_became_peered":"2024-03-26T04:10:55.570925+0000","last_unstale":"2024-03-26T04:11:18.163302+0000","last_undegraded":"2024-03-26T04:11:18.163302+0000","last_fullsized":"2024-03-26T04:11:18.163302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:31:12.320736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163685+0000","last_change":"2024-03-26T04:10:57.589808+0000","last_active":"2024-03-26T04:11:18.163685+0000","last_peered":"2024-03-26T04:11:18.163685+0000","last_clean":"2024-03-26T04:11:18.163685+0000","last_became_active":"2024-03-26T04:10:57.589435+0000","last_became_peered":"2024-03-26T04:10:57.589435+0000","last_unstale":"2024-03-26T04:11:18.163685+0000","last_undegraded":"2024-03-26T04:11:18.163685+0000","last_fullsized":"2024-03-26T04:11:18.163685+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:10:18.894370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165710+0000","last_change":"2024-03-26T04:10:51.542573+0000","last_active":"2024-03-26T04:11:18.165710+0000","last_peered":"2024-03-26T04:11:18.165710+0000","last_clean":"2024-03-26T04:11:18.165710+0000","last_became_active":"2024-03-26T04:10:51.542261+0000","last_became_peered":"2024-03-26T04:10:51.542261+0000","last_unstale":"2024-03-26T04:11:18.165710+0000","last_undegraded":"2024-03-26T04:11:18.165710+0000","last_fullsized":"2024-03-26T04:11:18.165710+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:24:19.907105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'36","reported_seq":93,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:21.588187+0000","last_change":"2024-03-26T04:10:53.558871+0000","last_active":"2024-03-26T04:11:21.588187+0000","last_peered":"2024-03-26T04:11:21.588187+0000","last_clean":"2024-03-26T04:11:21.588187+0000","last_became_active":"2024-03-26T04:10:53.558355+0000","last_became_peered":"2024-03-26T04:10:53.558355+0000","last_unstale":"2024-03-26T04:11:21.588187+0000","last_undegraded":"2024-03-26T04:11:21.588187+0000","last_fullsized":"2024-03-26T04:11:21.588187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:02:11.738180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:21.588356+0000","last_change":"2024-03-26T04:10:55.571065+0000","last_active":"2024-03-26T04:11:21.588356+0000","last_peered":"2024-03-26T04:11:21.588356+0000","last_clean":"2024-03-26T04:11:21.588356+0000","last_became_active":"2024-03-26T04:10:55.570483+0000","last_became_peered":"2024-03-26T04:10:55.570483+0000","last_unstale":"2024-03-26T04:11:21.588356+0000","last_undegraded":"2024-03-26T04:11:21.588356+0000","last_fullsized":"2024-03-26T04:11:21.588356+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:41:48.972497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162877+0000","last_change":"2024-03-26T04:10:57.591066+0000","last_active":"2024-03-26T04:11:18.162877+0000","last_peered":"2024-03-26T04:11:18.162877+0000","last_clean":"2024-03-26T04:11:18.162877+0000","last_became_active":"2024-03-26T04:10:57.590376+0000","last_became_peered":"2024-03-26T04:10:57.590376+0000","last_unstale":"2024-03-26T04:11:18.162877+0000","last_undegraded":"2024-03-26T04:11:18.162877+0000","last_fullsized":"2024-03-26T04:11:18.162877+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:32:17.365209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251217+0000","last_change":"2024-03-26T04:10:51.539809+0000","last_active":"2024-03-26T04:11:18.251217+0000","last_peered":"2024-03-26T04:11:18.251217+0000","last_clean":"2024-03-26T04:11:18.251217+0000","last_became_active":"2024-03-26T04:10:51.538894+0000","last_became_peered":"2024-03-26T04:10:51.538894+0000","last_unstale":"2024-03-26T04:11:18.251217+0000","last_undegraded":"2024-03-26T04:11:18.251217+0000","last_fullsized":"2024-03-26T04:11:18.251217+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:31:39.073532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251314+0000","last_change":"2024-03-26T04:10:53.558364+0000","last_active":"2024-03-26T04:11:18.251314+0000","last_peered":"2024-03-26T04:11:18.251314+0000","last_clean":"2024-03-26T04:11:18.251314+0000","last_became_active":"2024-03-26T04:10:53.557947+0000","last_became_peered":"2024-03-26T04:10:53.557947+0000","last_unstale":"2024-03-26T04:11:18.251314+0000","last_undegraded":"2024-03-26T04:11:18.251314+0000","last_fullsized":"2024-03-26T04:11:18.251314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:57:59.390390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:21.588153+0000","last_change":"2024-03-26T04:10:55.572260+0000","last_active":"2024-03-26T04:11:21.588153+0000","last_peered":"2024-03-26T04:11:21.588153+0000","last_clean":"2024-03-26T04:11:21.588153+0000","last_became_active":"2024-03-26T04:10:55.571933+0000","last_became_peered":"2024-03-26T04:10:55.571933+0000","last_unstale":"2024-03-26T04:11:21.588153+0000","last_undegraded":"2024-03-26T04:11:21.588153+0000","last_fullsized":"2024-03-26T04:11:21.588153+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:55:20.499657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162470+0000","last_change":"2024-03-26T04:10:57.585792+0000","last_active":"2024-03-26T04:11:18.162470+0000","last_peered":"2024-03-26T04:11:18.162470+0000","last_clean":"2024-03-26T04:11:18.162470+0000","last_became_active":"2024-03-26T04:10:57.585459+0000","last_became_peered":"2024-03-26T04:10:57.585459+0000","last_unstale":"2024-03-26T04:11:18.162470+0000","last_undegraded":"2024-03-26T04:11:18.162470+0000","last_fullsized":"2024-03-26T04:11:18.162470+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:21:14.735082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165487+0000","last_change":"2024-03-26T04:10:51.539331+0000","last_active":"2024-03-26T04:11:18.165487+0000","last_peered":"2024-03-26T04:11:18.165487+0000","last_clean":"2024-03-26T04:11:18.165487+0000","last_became_active":"2024-03-26T04:10:51.538879+0000","last_became_peered":"2024-03-26T04:10:51.538879+0000","last_unstale":"2024-03-26T04:11:18.165487+0000","last_undegraded":"2024-03-26T04:11:18.165487+0000","last_fullsized":"2024-03-26T04:11:18.165487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:18:04.320518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251143+0000","last_change":"2024-03-26T04:10:53.559627+0000","last_active":"2024-03-26T04:11:18.251143+0000","last_peered":"2024-03-26T04:11:18.251143+0000","last_clean":"2024-03-26T04:11:18.251143+0000","last_became_active":"2024-03-26T04:10:53.559292+0000","last_became_peered":"2024-03-26T04:10:53.559292+0000","last_unstale":"2024-03-26T04:11:18.251143+0000","last_undegraded":"2024-03-26T04:11:18.251143+0000","last_fullsized":"2024-03-26T04:11:18.251143+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:26:16.142401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162113+0000","last_change":"2024-03-26T04:10:55.572375+0000","last_active":"2024-03-26T04:11:18.162113+0000","last_peered":"2024-03-26T04:11:18.162113+0000","last_clean":"2024-03-26T04:11:18.162113+0000","last_became_active":"2024-03-26T04:10:55.570814+0000","last_became_peered":"2024-03-26T04:10:55.570814+0000","last_unstale":"2024-03-26T04:11:18.162113+0000","last_undegraded":"2024-03-26T04:11:18.162113+0000","last_fullsized":"2024-03-26T04:11:18.162113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:24:39.056159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164578+0000","last_change":"2024-03-26T04:10:57.591329+0000","last_active":"2024-03-26T04:11:18.164578+0000","last_peered":"2024-03-26T04:11:18.164578+0000","last_clean":"2024-03-26T04:11:18.164578+0000","last_became_active":"2024-03-26T04:10:57.590320+0000","last_became_peered":"2024-03-26T04:10:57.590320+0000","last_unstale":"2024-03-26T04:11:18.164578+0000","last_undegraded":"2024-03-26T04:11:18.164578+0000","last_fullsized":"2024-03-26T04:11:18.164578+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:35:02.778257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:21.588182+0000","last_change":"2024-03-26T04:10:55.575391+0000","last_active":"2024-03-26T04:11:21.588182+0000","last_peered":"2024-03-26T04:11:21.588182+0000","last_clean":"2024-03-26T04:11:21.588182+0000","last_became_active":"2024-03-26T04:10:55.574403+0000","last_became_peered":"2024-03-26T04:10:55.574403+0000","last_unstale":"2024-03-26T04:11:21.588182+0000","last_undegraded":"2024-03-26T04:11:21.588182+0000","last_fullsized":"2024-03-26T04:11:21.588182+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:31:27.389111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162614+0000","last_change":"2024-03-26T04:10:51.545944+0000","last_active":"2024-03-26T04:11:18.162614+0000","last_peered":"2024-03-26T04:11:18.162614+0000","last_clean":"2024-03-26T04:11:18.162614+0000","last_became_active":"2024-03-26T04:10:51.545625+0000","last_became_peered":"2024-03-26T04:10:51.545625+0000","last_unstale":"2024-03-26T04:11:18.162614+0000","last_undegraded":"2024-03-26T04:11:18.162614+0000","last_fullsized":"2024-03-26T04:11:18.162614+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:24:04.916197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162468+0000","last_change":"2024-03-26T04:10:53.561227+0000","last_active":"2024-03-26T04:11:18.162468+0000","last_peered":"2024-03-26T04:11:18.162468+0000","last_clean":"2024-03-26T04:11:18.162468+0000","last_became_active":"2024-03-26T04:10:53.560743+0000","last_became_peered":"2024-03-26T04:10:53.560743+0000","last_unstale":"2024-03-26T04:11:18.162468+0000","last_undegraded":"2024-03-26T04:11:18.162468+0000","last_fullsized":"2024-03-26T04:11:18.162468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:24:19.721900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163541+0000","last_change":"2024-03-26T04:10:57.586333+0000","last_active":"2024-03-26T04:11:18.163541+0000","last_peered":"2024-03-26T04:11:18.163541+0000","last_clean":"2024-03-26T04:11:18.163541+0000","last_became_active":"2024-03-26T04:10:57.586097+0000","last_became_peered":"2024-03-26T04:10:57.586097+0000","last_unstale":"2024-03-26T04:11:18.163541+0000","last_undegraded":"2024-03-26T04:11:18.163541+0000","last_fullsized":"2024-03-26T04:11:18.163541+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:57:39.245475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250620+0000","last_change":"2024-03-26T04:10:55.571111+0000","last_active":"2024-03-26T04:11:18.250620+0000","last_peered":"2024-03-26T04:11:18.250620+0000","last_clean":"2024-03-26T04:11:18.250620+0000","last_became_active":"2024-03-26T04:10:55.570718+0000","last_became_peered":"2024-03-26T04:10:55.570718+0000","last_unstale":"2024-03-26T04:11:18.250620+0000","last_undegraded":"2024-03-26T04:11:18.250620+0000","last_fullsized":"2024-03-26T04:11:18.250620+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:36:31.538712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248495+0000","last_change":"2024-03-26T04:10:51.542052+0000","last_active":"2024-03-26T04:11:18.248495+0000","last_peered":"2024-03-26T04:11:18.248495+0000","last_clean":"2024-03-26T04:11:18.248495+0000","last_became_active":"2024-03-26T04:10:51.540974+0000","last_became_peered":"2024-03-26T04:10:51.540974+0000","last_unstale":"2024-03-26T04:11:18.248495+0000","last_undegraded":"2024-03-26T04:11:18.248495+0000","last_fullsized":"2024-03-26T04:11:18.248495+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:51:47.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162232+0000","last_change":"2024-03-26T04:10:53.561000+0000","last_active":"2024-03-26T04:11:18.162232+0000","last_peered":"2024-03-26T04:11:18.162232+0000","last_clean":"2024-03-26T04:11:18.162232+0000","last_became_active":"2024-03-26T04:10:53.560703+0000","last_became_peered":"2024-03-26T04:10:53.560703+0000","last_unstale":"2024-03-26T04:11:18.162232+0000","last_undegraded":"2024-03-26T04:11:18.162232+0000","last_fullsized":"2024-03-26T04:11:18.162232+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:47:03.620113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165093+0000","last_change":"2024-03-26T04:10:57.585042+0000","last_active":"2024-03-26T04:11:18.165093+0000","last_peered":"2024-03-26T04:11:18.165093+0000","last_clean":"2024-03-26T04:11:18.165093+0000","last_became_active":"2024-03-26T04:10:57.584229+0000","last_became_peered":"2024-03-26T04:10:57.584229+0000","last_unstale":"2024-03-26T04:11:18.165093+0000","last_undegraded":"2024-03-26T04:11:18.165093+0000","last_fullsized":"2024-03-26T04:11:18.165093+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:01:21.799032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163317+0000","last_change":"2024-03-26T04:10:51.539367+0000","last_active":"2024-03-26T04:11:18.163317+0000","last_peered":"2024-03-26T04:11:18.163317+0000","last_clean":"2024-03-26T04:11:18.163317+0000","last_became_active":"2024-03-26T04:10:51.539050+0000","last_became_peered":"2024-03-26T04:10:51.539050+0000","last_unstale":"2024-03-26T04:11:18.163317+0000","last_undegraded":"2024-03-26T04:11:18.163317+0000","last_fullsized":"2024-03-26T04:11:18.163317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:30:52.553832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249964+0000","last_change":"2024-03-26T04:10:53.559236+0000","last_active":"2024-03-26T04:11:18.249964+0000","last_peered":"2024-03-26T04:11:18.249964+0000","last_clean":"2024-03-26T04:11:18.249964+0000","last_became_active":"2024-03-26T04:10:53.558749+0000","last_became_peered":"2024-03-26T04:10:53.558749+0000","last_unstale":"2024-03-26T04:11:18.249964+0000","last_undegraded":"2024-03-26T04:11:18.249964+0000","last_fullsized":"2024-03-26T04:11:18.249964+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:53:29.311641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163896+0000","last_change":"2024-03-26T04:10:55.575639+0000","last_active":"2024-03-26T04:11:18.163896+0000","last_peered":"2024-03-26T04:11:18.163896+0000","last_clean":"2024-03-26T04:11:18.163896+0000","last_became_active":"2024-03-26T04:10:55.574637+0000","last_became_peered":"2024-03-26T04:10:55.574637+0000","last_unstale":"2024-03-26T04:11:18.163896+0000","last_undegraded":"2024-03-26T04:11:18.163896+0000","last_fullsized":"2024-03-26T04:11:18.163896+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:43:55.496812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249385+0000","last_change":"2024-03-26T04:10:57.589110+0000","last_active":"2024-03-26T04:11:18.249385+0000","last_peered":"2024-03-26T04:11:18.249385+0000","last_clean":"2024-03-26T04:11:18.249385+0000","last_became_active":"2024-03-26T04:10:57.588387+0000","last_became_peered":"2024-03-26T04:10:57.588387+0000","last_unstale":"2024-03-26T04:11:18.249385+0000","last_undegraded":"2024-03-26T04:11:18.249385+0000","last_fullsized":"2024-03-26T04:11:18.249385+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:20:59.791679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:21.588238+0000","last_change":"2024-03-26T04:10:55.575876+0000","last_active":"2024-03-26T04:11:21.588238+0000","last_peered":"2024-03-26T04:11:21.588238+0000","last_clean":"2024-03-26T04:11:21.588238+0000","last_became_active":"2024-03-26T04:10:55.575509+0000","last_became_peered":"2024-03-26T04:10:55.575509+0000","last_unstale":"2024-03-26T04:11:21.588238+0000","last_undegraded":"2024-03-26T04:11:21.588238+0000","last_fullsized":"2024-03-26T04:11:21.588238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:08:53.905936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163542+0000","last_change":"2024-03-26T04:10:51.539899+0000","last_active":"2024-03-26T04:11:18.163542+0000","last_peered":"2024-03-26T04:11:18.163542+0000","last_clean":"2024-03-26T04:11:18.163542+0000","last_became_active":"2024-03-26T04:10:51.539497+0000","last_became_peered":"2024-03-26T04:10:51.539497+0000","last_unstale":"2024-03-26T04:11:18.163542+0000","last_undegraded":"2024-03-26T04:11:18.163542+0000","last_fullsized":"2024-03-26T04:11:18.163542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:09:06.666206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'25","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163412+0000","last_change":"2024-03-26T04:10:53.562993+0000","last_active":"2024-03-26T04:11:18.163412+0000","last_peered":"2024-03-26T04:11:18.163412+0000","last_clean":"2024-03-26T04:11:18.163412+0000","last_became_active":"2024-03-26T04:10:53.561927+0000","last_became_peered":"2024-03-26T04:10:53.561927+0000","last_unstale":"2024-03-26T04:11:18.163412+0000","last_undegraded":"2024-03-26T04:11:18.163412+0000","last_fullsized":"2024-03-26T04:11:18.163412+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:58:41.744418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162674+0000","last_change":"2024-03-26T04:10:57.584793+0000","last_active":"2024-03-26T04:11:18.162674+0000","last_peered":"2024-03-26T04:11:18.162674+0000","last_clean":"2024-03-26T04:11:18.162674+0000","last_became_active":"2024-03-26T04:10:57.584509+0000","last_became_peered":"2024-03-26T04:10:57.584509+0000","last_unstale":"2024-03-26T04:11:18.162674+0000","last_undegraded":"2024-03-26T04:11:18.162674+0000","last_fullsized":"2024-03-26T04:11:18.162674+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:55:26.088975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163248+0000","last_change":"2024-03-26T04:10:51.540059+0000","last_active":"2024-03-26T04:11:18.163248+0000","last_peered":"2024-03-26T04:11:18.163248+0000","last_clean":"2024-03-26T04:11:18.163248+0000","last_became_active":"2024-03-26T04:10:51.539604+0000","last_became_peered":"2024-03-26T04:10:51.539604+0000","last_unstale":"2024-03-26T04:11:18.163248+0000","last_undegraded":"2024-03-26T04:11:18.163248+0000","last_fullsized":"2024-03-26T04:11:18.163248+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:48:36.654899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164751+0000","last_change":"2024-03-26T04:10:53.560915+0000","last_active":"2024-03-26T04:11:18.164751+0000","last_peered":"2024-03-26T04:11:18.164751+0000","last_clean":"2024-03-26T04:11:18.164751+0000","last_became_active":"2024-03-26T04:10:53.560168+0000","last_became_peered":"2024-03-26T04:10:53.560168+0000","last_unstale":"2024-03-26T04:11:18.164751+0000","last_undegraded":"2024-03-26T04:11:18.164751+0000","last_fullsized":"2024-03-26T04:11:18.164751+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:57:11.873572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163206+0000","last_change":"2024-03-26T04:10:55.576228+0000","last_active":"2024-03-26T04:11:18.163206+0000","last_peered":"2024-03-26T04:11:18.163206+0000","last_clean":"2024-03-26T04:11:18.163206+0000","last_became_active":"2024-03-26T04:10:55.575736+0000","last_became_peered":"2024-03-26T04:10:55.575736+0000","last_unstale":"2024-03-26T04:11:18.163206+0000","last_undegraded":"2024-03-26T04:11:18.163206+0000","last_fullsized":"2024-03-26T04:11:18.163206+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:08:41.061088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248221+0000","last_change":"2024-03-26T04:10:57.589354+0000","last_active":"2024-03-26T04:11:18.248221+0000","last_peered":"2024-03-26T04:11:18.248221+0000","last_clean":"2024-03-26T04:11:18.248221+0000","last_became_active":"2024-03-26T04:10:57.588529+0000","last_became_peered":"2024-03-26T04:10:57.588529+0000","last_unstale":"2024-03-26T04:11:18.248221+0000","last_undegraded":"2024-03-26T04:11:18.248221+0000","last_fullsized":"2024-03-26T04:11:18.248221+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:07:23.322125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165440+0000","last_change":"2024-03-26T04:10:51.540146+0000","last_active":"2024-03-26T04:11:18.165440+0000","last_peered":"2024-03-26T04:11:18.165440+0000","last_clean":"2024-03-26T04:11:18.165440+0000","last_became_active":"2024-03-26T04:10:51.539833+0000","last_became_peered":"2024-03-26T04:10:51.539833+0000","last_unstale":"2024-03-26T04:11:18.165440+0000","last_undegraded":"2024-03-26T04:11:18.165440+0000","last_fullsized":"2024-03-26T04:11:18.165440+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:24:27.959472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164234+0000","last_change":"2024-03-26T04:10:53.561376+0000","last_active":"2024-03-26T04:11:18.164234+0000","last_peered":"2024-03-26T04:11:18.164234+0000","last_clean":"2024-03-26T04:11:18.164234+0000","last_became_active":"2024-03-26T04:10:53.560948+0000","last_became_peered":"2024-03-26T04:10:53.560948+0000","last_unstale":"2024-03-26T04:11:18.164234+0000","last_undegraded":"2024-03-26T04:11:18.164234+0000","last_fullsized":"2024-03-26T04:11:18.164234+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:31:33.079418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249363+0000","last_change":"2024-03-26T04:10:55.570508+0000","last_active":"2024-03-26T04:11:18.249363+0000","last_peered":"2024-03-26T04:11:18.249363+0000","last_clean":"2024-03-26T04:11:18.249363+0000","last_became_active":"2024-03-26T04:10:55.570063+0000","last_became_peered":"2024-03-26T04:10:55.570063+0000","last_unstale":"2024-03-26T04:11:18.249363+0000","last_undegraded":"2024-03-26T04:11:18.249363+0000","last_fullsized":"2024-03-26T04:11:18.249363+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:25:02.612299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162748+0000","last_change":"2024-03-26T04:10:57.591470+0000","last_active":"2024-03-26T04:11:18.162748+0000","last_peered":"2024-03-26T04:11:18.162748+0000","last_clean":"2024-03-26T04:11:18.162748+0000","last_became_active":"2024-03-26T04:10:57.591077+0000","last_became_peered":"2024-03-26T04:10:57.591077+0000","last_unstale":"2024-03-26T04:11:18.162748+0000","last_undegraded":"2024-03-26T04:11:18.162748+0000","last_fullsized":"2024-03-26T04:11:18.162748+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:42:40.920656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251052+0000","last_change":"2024-03-26T04:10:51.541457+0000","last_active":"2024-03-26T04:11:18.251052+0000","last_peered":"2024-03-26T04:11:18.251052+0000","last_clean":"2024-03-26T04:11:18.251052+0000","last_became_active":"2024-03-26T04:10:51.540871+0000","last_became_peered":"2024-03-26T04:10:51.540871+0000","last_unstale":"2024-03-26T04:11:18.251052+0000","last_undegraded":"2024-03-26T04:11:18.251052+0000","last_fullsized":"2024-03-26T04:11:18.251052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:51:16.523756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165587+0000","last_change":"2024-03-26T04:10:53.563631+0000","last_active":"2024-03-26T04:11:18.165587+0000","last_peered":"2024-03-26T04:11:18.165587+0000","last_clean":"2024-03-26T04:11:18.165587+0000","last_became_active":"2024-03-26T04:10:53.562681+0000","last_became_peered":"2024-03-26T04:10:53.562681+0000","last_unstale":"2024-03-26T04:11:18.165587+0000","last_undegraded":"2024-03-26T04:11:18.165587+0000","last_fullsized":"2024-03-26T04:11:18.165587+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:47:16.349668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162372+0000","last_change":"2024-03-26T04:10:55.570942+0000","last_active":"2024-03-26T04:11:18.162372+0000","last_peered":"2024-03-26T04:11:18.162372+0000","last_clean":"2024-03-26T04:11:18.162372+0000","last_became_active":"2024-03-26T04:10:55.570516+0000","last_became_peered":"2024-03-26T04:10:55.570516+0000","last_unstale":"2024-03-26T04:11:18.162372+0000","last_undegraded":"2024-03-26T04:11:18.162372+0000","last_fullsized":"2024-03-26T04:11:18.162372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:10:29.587494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411342,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:11:34.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:11:34.863 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:35.423 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:35 smithi136 ceph-mon[25848]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:11:35.487 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:35 smithi149 ceph-mon[30657]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:11:35.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:35 smithi204 ceph-mon[30652]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:11:36.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:36 smithi136 ceph-mon[25848]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T04:11:36.487 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:36 smithi149 ceph-mon[30657]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T04:11:36.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:36 smithi204 ceph-mon[30652]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T04:11:38.053 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:11:38.055 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-03-26T04:11:38.151 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:38 smithi136 ceph-mon[25848]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:38.487 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:38 smithi149 ceph-mon[30657]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:38.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:38 smithi204 ceph-mon[30652]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:39.042 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-26T04:11:36.175930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96624,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126544,"statfs":{"total":287972524032,"available":287873581056,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002622"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162609+0000","last_change":"2024-03-26T04:10:57.593375+0000","last_active":"2024-03-26T04:11:18.162609+0000","last_peered":"2024-03-26T04:11:18.162609+0000","last_clean":"2024-03-26T04:11:18.162609+0000","last_became_active":"2024-03-26T04:10:57.592779+0000","last_became_peered":"2024-03-26T04:10:57.592779+0000","last_unstale":"2024-03-26T04:11:18.162609+0000","last_undegraded":"2024-03-26T04:11:18.162609+0000","last_fullsized":"2024-03-26T04:11:18.162609+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:33:12.453911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.161923+0000","last_change":"2024-03-26T04:10:55.570493+0000","last_active":"2024-03-26T04:11:18.161923+0000","last_peered":"2024-03-26T04:11:18.161923+0000","last_clean":"2024-03-26T04:11:18.161923+0000","last_became_active":"2024-03-26T04:10:55.570031+0000","last_became_peered":"2024-03-26T04:10:55.570031+0000","last_unstale":"2024-03-26T04:11:18.161923+0000","last_undegraded":"2024-03-26T04:11:18.161923+0000","last_fullsized":"2024-03-26T04:11:18.161923+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:16:20.291852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250919+0000","last_change":"2024-03-26T04:10:51.541831+0000","last_active":"2024-03-26T04:11:18.250919+0000","last_peered":"2024-03-26T04:11:18.250919+0000","last_clean":"2024-03-26T04:11:18.250919+0000","last_became_active":"2024-03-26T04:10:51.541118+0000","last_became_peered":"2024-03-26T04:10:51.541118+0000","last_unstale":"2024-03-26T04:11:18.250919+0000","last_undegraded":"2024-03-26T04:11:18.250919+0000","last_fullsized":"2024-03-26T04:11:18.250919+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:39:42.843470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250994+0000","last_change":"2024-03-26T04:10:53.561664+0000","last_active":"2024-03-26T04:11:18.250994+0000","last_peered":"2024-03-26T04:11:18.250994+0000","last_clean":"2024-03-26T04:11:18.250994+0000","last_became_active":"2024-03-26T04:10:53.561090+0000","last_became_peered":"2024-03-26T04:10:53.561090+0000","last_unstale":"2024-03-26T04:11:18.250994+0000","last_undegraded":"2024-03-26T04:11:18.250994+0000","last_fullsized":"2024-03-26T04:11:18.250994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:19:14.305912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248459+0000","last_change":"2024-03-26T04:10:57.587673+0000","last_active":"2024-03-26T04:11:18.248459+0000","last_peered":"2024-03-26T04:11:18.248459+0000","last_clean":"2024-03-26T04:11:18.248459+0000","last_became_active":"2024-03-26T04:10:57.586424+0000","last_became_peered":"2024-03-26T04:10:57.586424+0000","last_unstale":"2024-03-26T04:11:18.248459+0000","last_undegraded":"2024-03-26T04:11:18.248459+0000","last_fullsized":"2024-03-26T04:11:18.248459+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:16:07.181924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163056+0000","last_change":"2024-03-26T04:10:55.576080+0000","last_active":"2024-03-26T04:11:18.163056+0000","last_peered":"2024-03-26T04:11:18.163056+0000","last_clean":"2024-03-26T04:11:18.163056+0000","last_became_active":"2024-03-26T04:10:55.575740+0000","last_became_peered":"2024-03-26T04:10:55.575740+0000","last_unstale":"2024-03-26T04:11:18.163056+0000","last_undegraded":"2024-03-26T04:11:18.163056+0000","last_fullsized":"2024-03-26T04:11:18.163056+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:26:16.418596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251328+0000","last_change":"2024-03-26T04:10:51.541668+0000","last_active":"2024-03-26T04:11:18.251328+0000","last_peered":"2024-03-26T04:11:18.251328+0000","last_clean":"2024-03-26T04:11:18.251328+0000","last_became_active":"2024-03-26T04:10:51.541122+0000","last_became_peered":"2024-03-26T04:10:51.541122+0000","last_unstale":"2024-03-26T04:11:18.251328+0000","last_undegraded":"2024-03-26T04:11:18.251328+0000","last_fullsized":"2024-03-26T04:11:18.251328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:06:03.006022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250720+0000","last_change":"2024-03-26T04:10:53.560066+0000","last_active":"2024-03-26T04:11:18.250720+0000","last_peered":"2024-03-26T04:11:18.250720+0000","last_clean":"2024-03-26T04:11:18.250720+0000","last_became_active":"2024-03-26T04:10:53.559704+0000","last_became_peered":"2024-03-26T04:10:53.559704+0000","last_unstale":"2024-03-26T04:11:18.250720+0000","last_undegraded":"2024-03-26T04:11:18.250720+0000","last_fullsized":"2024-03-26T04:11:18.250720+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:18:40.816885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162066+0000","last_change":"2024-03-26T04:10:57.585660+0000","last_active":"2024-03-26T04:11:18.162066+0000","last_peered":"2024-03-26T04:11:18.162066+0000","last_clean":"2024-03-26T04:11:18.162066+0000","last_became_active":"2024-03-26T04:10:57.585070+0000","last_became_peered":"2024-03-26T04:10:57.585070+0000","last_unstale":"2024-03-26T04:11:18.162066+0000","last_undegraded":"2024-03-26T04:11:18.162066+0000","last_fullsized":"2024-03-26T04:11:18.162066+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:52:21.766227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163790+0000","last_change":"2024-03-26T04:10:55.575317+0000","last_active":"2024-03-26T04:11:18.163790+0000","last_peered":"2024-03-26T04:11:18.163790+0000","last_clean":"2024-03-26T04:11:18.163790+0000","last_became_active":"2024-03-26T04:10:55.574475+0000","last_became_peered":"2024-03-26T04:10:55.574475+0000","last_unstale":"2024-03-26T04:11:18.163790+0000","last_undegraded":"2024-03-26T04:11:18.163790+0000","last_fullsized":"2024-03-26T04:11:18.163790+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:50:16.780131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165937+0000","last_change":"2024-03-26T04:10:51.542609+0000","last_active":"2024-03-26T04:11:18.165937+0000","last_peered":"2024-03-26T04:11:18.165937+0000","last_clean":"2024-03-26T04:11:18.165937+0000","last_became_active":"2024-03-26T04:10:51.542330+0000","last_became_peered":"2024-03-26T04:10:51.542330+0000","last_unstale":"2024-03-26T04:11:18.165937+0000","last_undegraded":"2024-03-26T04:11:18.165937+0000","last_fullsized":"2024-03-26T04:11:18.165937+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:02:41.765004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164082+0000","last_change":"2024-03-26T04:10:53.563395+0000","last_active":"2024-03-26T04:11:18.164082+0000","last_peered":"2024-03-26T04:11:18.164082+0000","last_clean":"2024-03-26T04:11:18.164082+0000","last_became_active":"2024-03-26T04:10:53.562987+0000","last_became_peered":"2024-03-26T04:10:53.562987+0000","last_unstale":"2024-03-26T04:11:18.164082+0000","last_undegraded":"2024-03-26T04:11:18.164082+0000","last_fullsized":"2024-03-26T04:11:18.164082+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:36:10.377852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163536+0000","last_change":"2024-03-26T04:10:57.592688+0000","last_active":"2024-03-26T04:11:18.163536+0000","last_peered":"2024-03-26T04:11:18.163536+0000","last_clean":"2024-03-26T04:11:18.163536+0000","last_became_active":"2024-03-26T04:10:57.591008+0000","last_became_peered":"2024-03-26T04:10:57.591008+0000","last_unstale":"2024-03-26T04:11:18.163536+0000","last_undegraded":"2024-03-26T04:11:18.163536+0000","last_fullsized":"2024-03-26T04:11:18.163536+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:18:08.773881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163043+0000","last_change":"2024-03-26T04:10:55.575606+0000","last_active":"2024-03-26T04:11:18.163043+0000","last_peered":"2024-03-26T04:11:18.163043+0000","last_clean":"2024-03-26T04:11:18.163043+0000","last_became_active":"2024-03-26T04:10:55.574646+0000","last_became_peered":"2024-03-26T04:10:55.574646+0000","last_unstale":"2024-03-26T04:11:18.163043+0000","last_undegraded":"2024-03-26T04:11:18.163043+0000","last_fullsized":"2024-03-26T04:11:18.163043+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:27:58.908247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163092+0000","last_change":"2024-03-26T04:10:51.539728+0000","last_active":"2024-03-26T04:11:18.163092+0000","last_peered":"2024-03-26T04:11:18.163092+0000","last_clean":"2024-03-26T04:11:18.163092+0000","last_became_active":"2024-03-26T04:10:51.539369+0000","last_became_peered":"2024-03-26T04:10:51.539369+0000","last_unstale":"2024-03-26T04:11:18.163092+0000","last_undegraded":"2024-03-26T04:11:18.163092+0000","last_fullsized":"2024-03-26T04:11:18.163092+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:19:44.450425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'20","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165819+0000","last_change":"2024-03-26T04:10:53.563880+0000","last_active":"2024-03-26T04:11:18.165819+0000","last_peered":"2024-03-26T04:11:18.165819+0000","last_clean":"2024-03-26T04:11:18.165819+0000","last_became_active":"2024-03-26T04:10:53.563564+0000","last_became_peered":"2024-03-26T04:10:53.563564+0000","last_unstale":"2024-03-26T04:11:18.165819+0000","last_undegraded":"2024-03-26T04:11:18.165819+0000","last_fullsized":"2024-03-26T04:11:18.165819+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:53:53.377906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163733+0000","last_change":"2024-03-26T04:10:57.589990+0000","last_active":"2024-03-26T04:11:18.163733+0000","last_peered":"2024-03-26T04:11:18.163733+0000","last_clean":"2024-03-26T04:11:18.163733+0000","last_became_active":"2024-03-26T04:10:57.589668+0000","last_became_peered":"2024-03-26T04:10:57.589668+0000","last_unstale":"2024-03-26T04:11:18.163733+0000","last_undegraded":"2024-03-26T04:11:18.163733+0000","last_fullsized":"2024-03-26T04:11:18.163733+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:37:03.809732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165191+0000","last_change":"2024-03-26T04:10:55.572450+0000","last_active":"2024-03-26T04:11:18.165191+0000","last_peered":"2024-03-26T04:11:18.165191+0000","last_clean":"2024-03-26T04:11:18.165191+0000","last_became_active":"2024-03-26T04:10:55.571394+0000","last_became_peered":"2024-03-26T04:10:55.571394+0000","last_unstale":"2024-03-26T04:11:18.165191+0000","last_undegraded":"2024-03-26T04:11:18.165191+0000","last_fullsized":"2024-03-26T04:11:18.165191+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:02:03.086578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163671+0000","last_change":"2024-03-26T04:10:51.540563+0000","last_active":"2024-03-26T04:11:18.163671+0000","last_peered":"2024-03-26T04:11:18.163671+0000","last_clean":"2024-03-26T04:11:18.163671+0000","last_became_active":"2024-03-26T04:10:51.540250+0000","last_became_peered":"2024-03-26T04:10:51.540250+0000","last_unstale":"2024-03-26T04:11:18.163671+0000","last_undegraded":"2024-03-26T04:11:18.163671+0000","last_fullsized":"2024-03-26T04:11:18.163671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:54:51.397311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163451+0000","last_change":"2024-03-26T04:10:53.562420+0000","last_active":"2024-03-26T04:11:18.163451+0000","last_peered":"2024-03-26T04:11:18.163451+0000","last_clean":"2024-03-26T04:11:18.163451+0000","last_became_active":"2024-03-26T04:10:53.562088+0000","last_became_peered":"2024-03-26T04:10:53.562088+0000","last_unstale":"2024-03-26T04:11:18.163451+0000","last_undegraded":"2024-03-26T04:11:18.163451+0000","last_fullsized":"2024-03-26T04:11:18.163451+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:42:56.273619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250065+0000","last_change":"2024-03-26T04:10:57.588378+0000","last_active":"2024-03-26T04:11:18.250065+0000","last_peered":"2024-03-26T04:11:18.250065+0000","last_clean":"2024-03-26T04:11:18.250065+0000","last_became_active":"2024-03-26T04:10:57.587052+0000","last_became_peered":"2024-03-26T04:10:57.587052+0000","last_unstale":"2024-03-26T04:11:18.250065+0000","last_undegraded":"2024-03-26T04:11:18.250065+0000","last_fullsized":"2024-03-26T04:11:18.250065+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:03:08.339152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163340+0000","last_change":"2024-03-26T04:10:55.572854+0000","last_active":"2024-03-26T04:11:18.163340+0000","last_peered":"2024-03-26T04:11:18.163340+0000","last_clean":"2024-03-26T04:11:18.163340+0000","last_became_active":"2024-03-26T04:10:55.572555+0000","last_became_peered":"2024-03-26T04:10:55.572555+0000","last_unstale":"2024-03-26T04:11:18.163340+0000","last_undegraded":"2024-03-26T04:11:18.163340+0000","last_fullsized":"2024-03-26T04:11:18.163340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:25:52.001859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163761+0000","last_change":"2024-03-26T04:10:51.545236+0000","last_active":"2024-03-26T04:11:18.163761+0000","last_peered":"2024-03-26T04:11:18.163761+0000","last_clean":"2024-03-26T04:11:18.163761+0000","last_became_active":"2024-03-26T04:10:51.544876+0000","last_became_peered":"2024-03-26T04:10:51.544876+0000","last_unstale":"2024-03-26T04:11:18.163761+0000","last_undegraded":"2024-03-26T04:11:18.163761+0000","last_fullsized":"2024-03-26T04:11:18.163761+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:20:04.457577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163427+0000","last_change":"2024-03-26T04:10:53.561304+0000","last_active":"2024-03-26T04:11:18.163427+0000","last_peered":"2024-03-26T04:11:18.163427+0000","last_clean":"2024-03-26T04:11:18.163427+0000","last_became_active":"2024-03-26T04:10:53.561003+0000","last_became_peered":"2024-03-26T04:10:53.561003+0000","last_unstale":"2024-03-26T04:11:18.163427+0000","last_undegraded":"2024-03-26T04:11:18.163427+0000","last_fullsized":"2024-03-26T04:11:18.163427+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:33:51.344073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249771+0000","last_change":"2024-03-26T04:10:57.589417+0000","last_active":"2024-03-26T04:11:18.249771+0000","last_peered":"2024-03-26T04:11:18.249771+0000","last_clean":"2024-03-26T04:11:18.249771+0000","last_became_active":"2024-03-26T04:10:57.588537+0000","last_became_peered":"2024-03-26T04:10:57.588537+0000","last_unstale":"2024-03-26T04:11:18.249771+0000","last_undegraded":"2024-03-26T04:11:18.249771+0000","last_fullsized":"2024-03-26T04:11:18.249771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:42:01.264561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:26.589650+0000","last_change":"2024-03-26T04:10:55.572583+0000","last_active":"2024-03-26T04:11:26.589650+0000","last_peered":"2024-03-26T04:11:26.589650+0000","last_clean":"2024-03-26T04:11:26.589650+0000","last_became_active":"2024-03-26T04:10:55.571273+0000","last_became_peered":"2024-03-26T04:10:55.571273+0000","last_unstale":"2024-03-26T04:11:26.589650+0000","last_undegraded":"2024-03-26T04:11:26.589650+0000","last_fullsized":"2024-03-26T04:11:26.589650+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:51:35.178605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248646+0000","last_change":"2024-03-26T04:10:51.542935+0000","last_active":"2024-03-26T04:11:18.248646+0000","last_peered":"2024-03-26T04:11:18.248646+0000","last_clean":"2024-03-26T04:11:18.248646+0000","last_became_active":"2024-03-26T04:10:51.541850+0000","last_became_peered":"2024-03-26T04:10:51.541850+0000","last_unstale":"2024-03-26T04:11:18.248646+0000","last_undegraded":"2024-03-26T04:11:18.248646+0000","last_fullsized":"2024-03-26T04:11:18.248646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:29:40.572990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248140+0000","last_change":"2024-03-26T04:10:53.561899+0000","last_active":"2024-03-26T04:11:18.248140+0000","last_peered":"2024-03-26T04:11:18.248140+0000","last_clean":"2024-03-26T04:11:18.248140+0000","last_became_active":"2024-03-26T04:10:53.561177+0000","last_became_peered":"2024-03-26T04:10:53.561177+0000","last_unstale":"2024-03-26T04:11:18.248140+0000","last_undegraded":"2024-03-26T04:11:18.248140+0000","last_fullsized":"2024-03-26T04:11:18.248140+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:17:38.210701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164473+0000","last_change":"2024-03-26T04:10:57.592801+0000","last_active":"2024-03-26T04:11:18.164473+0000","last_peered":"2024-03-26T04:11:18.164473+0000","last_clean":"2024-03-26T04:11:18.164473+0000","last_became_active":"2024-03-26T04:10:57.591597+0000","last_became_peered":"2024-03-26T04:10:57.591597+0000","last_unstale":"2024-03-26T04:11:18.164473+0000","last_undegraded":"2024-03-26T04:11:18.164473+0000","last_fullsized":"2024-03-26T04:11:18.164473+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:34:50.321529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250159+0000","last_change":"2024-03-26T04:10:55.570574+0000","last_active":"2024-03-26T04:11:18.250159+0000","last_peered":"2024-03-26T04:11:18.250159+0000","last_clean":"2024-03-26T04:11:18.250159+0000","last_became_active":"2024-03-26T04:10:55.570234+0000","last_became_peered":"2024-03-26T04:10:55.570234+0000","last_unstale":"2024-03-26T04:11:18.250159+0000","last_undegraded":"2024-03-26T04:11:18.250159+0000","last_fullsized":"2024-03-26T04:11:18.250159+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:18:44.535190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162074+0000","last_change":"2024-03-26T04:10:51.543502+0000","last_active":"2024-03-26T04:11:18.162074+0000","last_peered":"2024-03-26T04:11:18.162074+0000","last_clean":"2024-03-26T04:11:18.162074+0000","last_became_active":"2024-03-26T04:10:51.542690+0000","last_became_peered":"2024-03-26T04:10:51.542690+0000","last_unstale":"2024-03-26T04:11:18.162074+0000","last_undegraded":"2024-03-26T04:11:18.162074+0000","last_fullsized":"2024-03-26T04:11:18.162074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:27:12.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'39","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248648+0000","last_change":"2024-03-26T04:10:53.559877+0000","last_active":"2024-03-26T04:11:18.248648+0000","last_peered":"2024-03-26T04:11:18.248648+0000","last_clean":"2024-03-26T04:11:18.248648+0000","last_became_active":"2024-03-26T04:10:53.559481+0000","last_became_peered":"2024-03-26T04:10:53.559481+0000","last_unstale":"2024-03-26T04:11:18.248648+0000","last_undegraded":"2024-03-26T04:11:18.248648+0000","last_fullsized":"2024-03-26T04:11:18.248648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:54:01.903188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163214+0000","last_change":"2024-03-26T04:10:57.592925+0000","last_active":"2024-03-26T04:11:18.163214+0000","last_peered":"2024-03-26T04:11:18.163214+0000","last_clean":"2024-03-26T04:11:18.163214+0000","last_became_active":"2024-03-26T04:10:57.591484+0000","last_became_peered":"2024-03-26T04:10:57.591484+0000","last_unstale":"2024-03-26T04:11:18.163214+0000","last_undegraded":"2024-03-26T04:11:18.163214+0000","last_fullsized":"2024-03-26T04:11:18.163214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:18.678133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248340+0000","last_change":"2024-03-26T04:10:55.570968+0000","last_active":"2024-03-26T04:11:18.248340+0000","last_peered":"2024-03-26T04:11:18.248340+0000","last_clean":"2024-03-26T04:11:18.248340+0000","last_became_active":"2024-03-26T04:10:55.570332+0000","last_became_peered":"2024-03-26T04:10:55.570332+0000","last_unstale":"2024-03-26T04:11:18.248340+0000","last_undegraded":"2024-03-26T04:11:18.248340+0000","last_fullsized":"2024-03-26T04:11:18.248340+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:07:43.244996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165698+0000","last_change":"2024-03-26T04:10:51.545345+0000","last_active":"2024-03-26T04:11:18.165698+0000","last_peered":"2024-03-26T04:11:18.165698+0000","last_clean":"2024-03-26T04:11:18.165698+0000","last_became_active":"2024-03-26T04:10:51.545000+0000","last_became_peered":"2024-03-26T04:10:51.545000+0000","last_unstale":"2024-03-26T04:11:18.165698+0000","last_undegraded":"2024-03-26T04:11:18.165698+0000","last_fullsized":"2024-03-26T04:11:18.165698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:16:21.191400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'22","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165791+0000","last_change":"2024-03-26T04:10:53.562655+0000","last_active":"2024-03-26T04:11:18.165791+0000","last_peered":"2024-03-26T04:11:18.165791+0000","last_clean":"2024-03-26T04:11:18.165791+0000","last_became_active":"2024-03-26T04:10:53.561153+0000","last_became_peered":"2024-03-26T04:10:53.561153+0000","last_unstale":"2024-03-26T04:11:18.165791+0000","last_undegraded":"2024-03-26T04:11:18.165791+0000","last_fullsized":"2024-03-26T04:11:18.165791+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:45:11.285844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163148+0000","last_change":"2024-03-26T04:10:57.589809+0000","last_active":"2024-03-26T04:11:18.163148+0000","last_peered":"2024-03-26T04:11:18.163148+0000","last_clean":"2024-03-26T04:11:18.163148+0000","last_became_active":"2024-03-26T04:10:57.589504+0000","last_became_peered":"2024-03-26T04:10:57.589504+0000","last_unstale":"2024-03-26T04:11:18.163148+0000","last_undegraded":"2024-03-26T04:11:18.163148+0000","last_fullsized":"2024-03-26T04:11:18.163148+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:07:07.029474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248564+0000","last_change":"2024-03-26T04:10:55.570631+0000","last_active":"2024-03-26T04:11:18.248564+0000","last_peered":"2024-03-26T04:11:18.248564+0000","last_clean":"2024-03-26T04:11:18.248564+0000","last_became_active":"2024-03-26T04:10:55.570198+0000","last_became_peered":"2024-03-26T04:10:55.570198+0000","last_unstale":"2024-03-26T04:11:18.248564+0000","last_undegraded":"2024-03-26T04:11:18.248564+0000","last_fullsized":"2024-03-26T04:11:18.248564+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:16:43.055656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165491+0000","last_change":"2024-03-26T04:10:51.542723+0000","last_active":"2024-03-26T04:11:18.165491+0000","last_peered":"2024-03-26T04:11:18.165491+0000","last_clean":"2024-03-26T04:11:18.165491+0000","last_became_active":"2024-03-26T04:10:51.542430+0000","last_became_peered":"2024-03-26T04:10:51.542430+0000","last_unstale":"2024-03-26T04:11:18.165491+0000","last_undegraded":"2024-03-26T04:11:18.165491+0000","last_fullsized":"2024-03-26T04:11:18.165491+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:34:46.065266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251159+0000","last_change":"2024-03-26T04:10:53.561834+0000","last_active":"2024-03-26T04:11:18.251159+0000","last_peered":"2024-03-26T04:11:18.251159+0000","last_clean":"2024-03-26T04:11:18.251159+0000","last_became_active":"2024-03-26T04:10:53.561132+0000","last_became_peered":"2024-03-26T04:10:53.561132+0000","last_unstale":"2024-03-26T04:11:18.251159+0000","last_undegraded":"2024-03-26T04:11:18.251159+0000","last_fullsized":"2024-03-26T04:11:18.251159+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:29:31.297404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162357+0000","last_change":"2024-03-26T04:10:57.585780+0000","last_active":"2024-03-26T04:11:18.162357+0000","last_peered":"2024-03-26T04:11:18.162357+0000","last_clean":"2024-03-26T04:11:18.162357+0000","last_became_active":"2024-03-26T04:10:57.585489+0000","last_became_peered":"2024-03-26T04:10:57.585489+0000","last_unstale":"2024-03-26T04:11:18.162357+0000","last_undegraded":"2024-03-26T04:11:18.162357+0000","last_fullsized":"2024-03-26T04:11:18.162357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:37:09.101121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164366+0000","last_change":"2024-03-26T04:10:55.575328+0000","last_active":"2024-03-26T04:11:18.164366+0000","last_peered":"2024-03-26T04:11:18.164366+0000","last_clean":"2024-03-26T04:11:18.164366+0000","last_became_active":"2024-03-26T04:10:55.574415+0000","last_became_peered":"2024-03-26T04:10:55.574415+0000","last_unstale":"2024-03-26T04:11:18.164366+0000","last_undegraded":"2024-03-26T04:11:18.164366+0000","last_fullsized":"2024-03-26T04:11:18.164366+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:06.530199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165773+0000","last_change":"2024-03-26T04:10:51.541500+0000","last_active":"2024-03-26T04:11:18.165773+0000","last_peered":"2024-03-26T04:11:18.165773+0000","last_clean":"2024-03-26T04:11:18.165773+0000","last_became_active":"2024-03-26T04:10:51.541204+0000","last_became_peered":"2024-03-26T04:10:51.541204+0000","last_unstale":"2024-03-26T04:11:18.165773+0000","last_undegraded":"2024-03-26T04:11:18.165773+0000","last_fullsized":"2024-03-26T04:11:18.165773+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:15:24.926305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164268+0000","last_change":"2024-03-26T04:10:53.561793+0000","last_active":"2024-03-26T04:11:18.164268+0000","last_peered":"2024-03-26T04:11:18.164268+0000","last_clean":"2024-03-26T04:11:18.164268+0000","last_became_active":"2024-03-26T04:10:53.561017+0000","last_became_peered":"2024-03-26T04:10:53.561017+0000","last_unstale":"2024-03-26T04:11:18.164268+0000","last_undegraded":"2024-03-26T04:11:18.164268+0000","last_fullsized":"2024-03-26T04:11:18.164268+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:31:24.536019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162569+0000","last_change":"2024-03-26T04:10:57.585277+0000","last_active":"2024-03-26T04:11:18.162569+0000","last_peered":"2024-03-26T04:11:18.162569+0000","last_clean":"2024-03-26T04:11:18.162569+0000","last_became_active":"2024-03-26T04:10:57.584678+0000","last_became_peered":"2024-03-26T04:10:57.584678+0000","last_unstale":"2024-03-26T04:11:18.162569+0000","last_undegraded":"2024-03-26T04:11:18.162569+0000","last_fullsized":"2024-03-26T04:11:18.162569+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:44:26.960546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249009+0000","last_change":"2024-03-26T04:10:55.569107+0000","last_active":"2024-03-26T04:11:18.249009+0000","last_peered":"2024-03-26T04:11:18.249009+0000","last_clean":"2024-03-26T04:11:18.249009+0000","last_became_active":"2024-03-26T04:10:55.568739+0000","last_became_peered":"2024-03-26T04:10:55.568739+0000","last_unstale":"2024-03-26T04:11:18.249009+0000","last_undegraded":"2024-03-26T04:11:18.249009+0000","last_fullsized":"2024-03-26T04:11:18.249009+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:42:27.416428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164173+0000","last_change":"2024-03-26T04:10:51.545202+0000","last_active":"2024-03-26T04:11:18.164173+0000","last_peered":"2024-03-26T04:11:18.164173+0000","last_clean":"2024-03-26T04:11:18.164173+0000","last_became_active":"2024-03-26T04:10:51.544915+0000","last_became_peered":"2024-03-26T04:10:51.544915+0000","last_unstale":"2024-03-26T04:11:18.164173+0000","last_undegraded":"2024-03-26T04:11:18.164173+0000","last_fullsized":"2024-03-26T04:11:18.164173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:18:13.441979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165526+0000","last_change":"2024-03-26T04:10:53.562613+0000","last_active":"2024-03-26T04:11:18.165526+0000","last_peered":"2024-03-26T04:11:18.165526+0000","last_clean":"2024-03-26T04:11:18.165526+0000","last_became_active":"2024-03-26T04:10:53.562311+0000","last_became_peered":"2024-03-26T04:10:53.562311+0000","last_unstale":"2024-03-26T04:11:18.165526+0000","last_undegraded":"2024-03-26T04:11:18.165526+0000","last_fullsized":"2024-03-26T04:11:18.165526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:34:23.272447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249672+0000","last_change":"2024-03-26T04:10:57.587709+0000","last_active":"2024-03-26T04:11:18.249672+0000","last_peered":"2024-03-26T04:11:18.249672+0000","last_clean":"2024-03-26T04:11:18.249672+0000","last_became_active":"2024-03-26T04:10:57.587117+0000","last_became_peered":"2024-03-26T04:10:57.587117+0000","last_unstale":"2024-03-26T04:11:18.249672+0000","last_undegraded":"2024-03-26T04:11:18.249672+0000","last_fullsized":"2024-03-26T04:11:18.249672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:03:34.379361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:26.589724+0000","last_change":"2024-03-26T04:10:55.573195+0000","last_active":"2024-03-26T04:11:26.589724+0000","last_peered":"2024-03-26T04:11:26.589724+0000","last_clean":"2024-03-26T04:11:26.589724+0000","last_became_active":"2024-03-26T04:10:55.571369+0000","last_became_peered":"2024-03-26T04:10:55.571369+0000","last_unstale":"2024-03-26T04:11:26.589724+0000","last_undegraded":"2024-03-26T04:11:26.589724+0000","last_fullsized":"2024-03-26T04:11:26.589724+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:32:10.267637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163202+0000","last_change":"2024-03-26T04:10:51.541539+0000","last_active":"2024-03-26T04:11:18.163202+0000","last_peered":"2024-03-26T04:11:18.163202+0000","last_clean":"2024-03-26T04:11:18.163202+0000","last_became_active":"2024-03-26T04:10:51.541247+0000","last_became_peered":"2024-03-26T04:10:51.541247+0000","last_unstale":"2024-03-26T04:11:18.163202+0000","last_undegraded":"2024-03-26T04:11:18.163202+0000","last_fullsized":"2024-03-26T04:11:18.163202+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:13:48.617563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249396+0000","last_change":"2024-03-26T04:10:53.558121+0000","last_active":"2024-03-26T04:11:18.249396+0000","last_peered":"2024-03-26T04:11:18.249396+0000","last_clean":"2024-03-26T04:11:18.249396+0000","last_became_active":"2024-03-26T04:10:53.557621+0000","last_became_peered":"2024-03-26T04:10:53.557621+0000","last_unstale":"2024-03-26T04:11:18.249396+0000","last_undegraded":"2024-03-26T04:11:18.249396+0000","last_fullsized":"2024-03-26T04:11:18.249396+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:28:39.538916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163924+0000","last_change":"2024-03-26T04:10:57.593608+0000","last_active":"2024-03-26T04:11:18.163924+0000","last_peered":"2024-03-26T04:11:18.163924+0000","last_clean":"2024-03-26T04:11:18.163924+0000","last_became_active":"2024-03-26T04:10:57.593262+0000","last_became_peered":"2024-03-26T04:10:57.593262+0000","last_unstale":"2024-03-26T04:11:18.163924+0000","last_undegraded":"2024-03-26T04:11:18.163924+0000","last_fullsized":"2024-03-26T04:11:18.163924+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:55:40.547438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:26.589735+0000","last_change":"2024-03-26T04:10:55.571420+0000","last_active":"2024-03-26T04:11:26.589735+0000","last_peered":"2024-03-26T04:11:26.589735+0000","last_clean":"2024-03-26T04:11:26.589735+0000","last_became_active":"2024-03-26T04:10:55.571113+0000","last_became_peered":"2024-03-26T04:10:55.571113+0000","last_unstale":"2024-03-26T04:11:26.589735+0000","last_undegraded":"2024-03-26T04:11:26.589735+0000","last_fullsized":"2024-03-26T04:11:26.589735+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:52:24.795919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162741+0000","last_change":"2024-03-26T04:10:51.539172+0000","last_active":"2024-03-26T04:11:18.162741+0000","last_peered":"2024-03-26T04:11:18.162741+0000","last_clean":"2024-03-26T04:11:18.162741+0000","last_became_active":"2024-03-26T04:10:51.538874+0000","last_became_peered":"2024-03-26T04:10:51.538874+0000","last_unstale":"2024-03-26T04:11:18.162741+0000","last_undegraded":"2024-03-26T04:11:18.162741+0000","last_fullsized":"2024-03-26T04:11:18.162741+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:34:25.716285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164312+0000","last_change":"2024-03-26T04:10:53.564494+0000","last_active":"2024-03-26T04:11:18.164312+0000","last_peered":"2024-03-26T04:11:18.164312+0000","last_clean":"2024-03-26T04:11:18.164312+0000","last_became_active":"2024-03-26T04:10:53.563661+0000","last_became_peered":"2024-03-26T04:10:53.563661+0000","last_unstale":"2024-03-26T04:11:18.164312+0000","last_undegraded":"2024-03-26T04:11:18.164312+0000","last_fullsized":"2024-03-26T04:11:18.164312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:14:49.234160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164457+0000","last_change":"2024-03-26T04:10:57.593462+0000","last_active":"2024-03-26T04:11:18.164457+0000","last_peered":"2024-03-26T04:11:18.164457+0000","last_clean":"2024-03-26T04:11:18.164457+0000","last_became_active":"2024-03-26T04:10:57.593153+0000","last_became_peered":"2024-03-26T04:10:57.593153+0000","last_unstale":"2024-03-26T04:11:18.164457+0000","last_undegraded":"2024-03-26T04:11:18.164457+0000","last_fullsized":"2024-03-26T04:11:18.164457+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:27:23.253305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251143+0000","last_change":"2024-03-26T04:10:55.572211+0000","last_active":"2024-03-26T04:11:18.251143+0000","last_peered":"2024-03-26T04:11:18.251143+0000","last_clean":"2024-03-26T04:11:18.251143+0000","last_became_active":"2024-03-26T04:10:55.571085+0000","last_became_peered":"2024-03-26T04:10:55.571085+0000","last_unstale":"2024-03-26T04:11:18.251143+0000","last_undegraded":"2024-03-26T04:11:18.251143+0000","last_fullsized":"2024-03-26T04:11:18.251143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:31:48.287587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163414+0000","last_change":"2024-03-26T04:10:51.545279+0000","last_active":"2024-03-26T04:11:18.163414+0000","last_peered":"2024-03-26T04:11:18.163414+0000","last_clean":"2024-03-26T04:11:18.163414+0000","last_became_active":"2024-03-26T04:10:51.544891+0000","last_became_peered":"2024-03-26T04:10:51.544891+0000","last_unstale":"2024-03-26T04:11:18.163414+0000","last_undegraded":"2024-03-26T04:11:18.163414+0000","last_fullsized":"2024-03-26T04:11:18.163414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:20:04.302284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162816+0000","last_change":"2024-03-26T04:10:53.565015+0000","last_active":"2024-03-26T04:11:18.162816+0000","last_peered":"2024-03-26T04:11:18.162816+0000","last_clean":"2024-03-26T04:11:18.162816+0000","last_became_active":"2024-03-26T04:10:53.564631+0000","last_became_peered":"2024-03-26T04:10:53.564631+0000","last_unstale":"2024-03-26T04:11:18.162816+0000","last_undegraded":"2024-03-26T04:11:18.162816+0000","last_fullsized":"2024-03-26T04:11:18.162816+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:31:27.784195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249667+0000","last_change":"2024-03-26T04:10:57.587248+0000","last_active":"2024-03-26T04:11:18.249667+0000","last_peered":"2024-03-26T04:11:18.249667+0000","last_clean":"2024-03-26T04:11:18.249667+0000","last_became_active":"2024-03-26T04:10:57.586726+0000","last_became_peered":"2024-03-26T04:10:57.586726+0000","last_unstale":"2024-03-26T04:11:18.249667+0000","last_undegraded":"2024-03-26T04:11:18.249667+0000","last_fullsized":"2024-03-26T04:11:18.249667+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:22:31.701143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250183+0000","last_change":"2024-03-26T04:10:55.570650+0000","last_active":"2024-03-26T04:11:18.250183+0000","last_peered":"2024-03-26T04:11:18.250183+0000","last_clean":"2024-03-26T04:11:18.250183+0000","last_became_active":"2024-03-26T04:10:55.569000+0000","last_became_peered":"2024-03-26T04:10:55.569000+0000","last_unstale":"2024-03-26T04:11:18.250183+0000","last_undegraded":"2024-03-26T04:11:18.250183+0000","last_fullsized":"2024-03-26T04:11:18.250183+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:31:34.623827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162207+0000","last_change":"2024-03-26T04:10:51.542655+0000","last_active":"2024-03-26T04:11:18.162207+0000","last_peered":"2024-03-26T04:11:18.162207+0000","last_clean":"2024-03-26T04:11:18.162207+0000","last_became_active":"2024-03-26T04:10:51.542153+0000","last_became_peered":"2024-03-26T04:10:51.542153+0000","last_unstale":"2024-03-26T04:11:18.162207+0000","last_undegraded":"2024-03-26T04:11:18.162207+0000","last_fullsized":"2024-03-26T04:11:18.162207+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:08:20.158401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162801+0000","last_change":"2024-03-26T04:10:53.561470+0000","last_active":"2024-03-26T04:11:18.162801+0000","last_peered":"2024-03-26T04:11:18.162801+0000","last_clean":"2024-03-26T04:11:18.162801+0000","last_became_active":"2024-03-26T04:10:53.560843+0000","last_became_peered":"2024-03-26T04:10:53.560843+0000","last_unstale":"2024-03-26T04:11:18.162801+0000","last_undegraded":"2024-03-26T04:11:18.162801+0000","last_fullsized":"2024-03-26T04:11:18.162801+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:30:17.446361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162473+0000","last_change":"2024-03-26T04:10:57.585543+0000","last_active":"2024-03-26T04:11:18.162473+0000","last_peered":"2024-03-26T04:11:18.162473+0000","last_clean":"2024-03-26T04:11:18.162473+0000","last_became_active":"2024-03-26T04:10:57.584803+0000","last_became_peered":"2024-03-26T04:10:57.584803+0000","last_unstale":"2024-03-26T04:11:18.162473+0000","last_undegraded":"2024-03-26T04:11:18.162473+0000","last_fullsized":"2024-03-26T04:11:18.162473+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:05:14.911209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164790+0000","last_change":"2024-03-26T04:10:51.541599+0000","last_active":"2024-03-26T04:11:18.164790+0000","last_peered":"2024-03-26T04:11:18.164790+0000","last_clean":"2024-03-26T04:11:18.164790+0000","last_became_active":"2024-03-26T04:10:51.541262+0000","last_became_peered":"2024-03-26T04:10:51.541262+0000","last_unstale":"2024-03-26T04:11:18.164790+0000","last_undegraded":"2024-03-26T04:11:18.164790+0000","last_fullsized":"2024-03-26T04:11:18.164790+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:57:13.317922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163906+0000","last_change":"2024-03-26T04:10:53.562649+0000","last_active":"2024-03-26T04:11:18.163906+0000","last_peered":"2024-03-26T04:11:18.163906+0000","last_clean":"2024-03-26T04:11:18.163906+0000","last_became_active":"2024-03-26T04:10:53.562347+0000","last_became_peered":"2024-03-26T04:10:53.562347+0000","last_unstale":"2024-03-26T04:11:18.163906+0000","last_undegraded":"2024-03-26T04:11:18.163906+0000","last_fullsized":"2024-03-26T04:11:18.163906+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:10:08.481378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:26.589798+0000","last_change":"2024-03-26T04:10:55.570854+0000","last_active":"2024-03-26T04:11:26.589798+0000","last_peered":"2024-03-26T04:11:26.589798+0000","last_clean":"2024-03-26T04:11:26.589798+0000","last_became_active":"2024-03-26T04:10:55.570398+0000","last_became_peered":"2024-03-26T04:10:55.570398+0000","last_unstale":"2024-03-26T04:11:26.589798+0000","last_undegraded":"2024-03-26T04:11:26.589798+0000","last_fullsized":"2024-03-26T04:11:26.589798+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:19:35.228456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249246+0000","last_change":"2024-03-26T04:10:57.587403+0000","last_active":"2024-03-26T04:11:18.249246+0000","last_peered":"2024-03-26T04:11:18.249246+0000","last_clean":"2024-03-26T04:11:18.249246+0000","last_became_active":"2024-03-26T04:10:57.586769+0000","last_became_peered":"2024-03-26T04:10:57.586769+0000","last_unstale":"2024-03-26T04:11:18.249246+0000","last_undegraded":"2024-03-26T04:11:18.249246+0000","last_fullsized":"2024-03-26T04:11:18.249246+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:24:57.514820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250714+0000","last_change":"2024-03-26T04:10:51.541556+0000","last_active":"2024-03-26T04:11:18.250714+0000","last_peered":"2024-03-26T04:11:18.250714+0000","last_clean":"2024-03-26T04:11:18.250714+0000","last_became_active":"2024-03-26T04:10:51.540912+0000","last_became_peered":"2024-03-26T04:10:51.540912+0000","last_unstale":"2024-03-26T04:11:18.250714+0000","last_undegraded":"2024-03-26T04:11:18.250714+0000","last_fullsized":"2024-03-26T04:11:18.250714+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:50:09.093505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:26.589655+0000","last_change":"2024-03-26T04:10:53.561394+0000","last_active":"2024-03-26T04:11:26.589655+0000","last_peered":"2024-03-26T04:11:26.589655+0000","last_clean":"2024-03-26T04:11:26.589655+0000","last_became_active":"2024-03-26T04:10:53.560920+0000","last_became_peered":"2024-03-26T04:10:53.560920+0000","last_unstale":"2024-03-26T04:11:26.589655+0000","last_undegraded":"2024-03-26T04:11:26.589655+0000","last_fullsized":"2024-03-26T04:11:26.589655+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:32:30.124529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162462+0000","last_change":"2024-03-26T04:10:55.570775+0000","last_active":"2024-03-26T04:11:18.162462+0000","last_peered":"2024-03-26T04:11:18.162462+0000","last_clean":"2024-03-26T04:11:18.162462+0000","last_became_active":"2024-03-26T04:10:55.570451+0000","last_became_peered":"2024-03-26T04:10:55.570451+0000","last_unstale":"2024-03-26T04:11:18.162462+0000","last_undegraded":"2024-03-26T04:11:18.162462+0000","last_fullsized":"2024-03-26T04:11:18.162462+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:50.696483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249498+0000","last_change":"2024-03-26T04:10:57.589035+0000","last_active":"2024-03-26T04:11:18.249498+0000","last_peered":"2024-03-26T04:11:18.249498+0000","last_clean":"2024-03-26T04:11:18.249498+0000","last_became_active":"2024-03-26T04:10:57.587980+0000","last_became_peered":"2024-03-26T04:10:57.587980+0000","last_unstale":"2024-03-26T04:11:18.249498+0000","last_undegraded":"2024-03-26T04:11:18.249498+0000","last_fullsized":"2024-03-26T04:11:18.249498+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:59.983185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163672+0000","last_change":"2024-03-26T04:10:51.545447+0000","last_active":"2024-03-26T04:11:18.163672+0000","last_peered":"2024-03-26T04:11:18.163672+0000","last_clean":"2024-03-26T04:11:18.163672+0000","last_became_active":"2024-03-26T04:10:51.545063+0000","last_became_peered":"2024-03-26T04:10:51.545063+0000","last_unstale":"2024-03-26T04:11:18.163672+0000","last_undegraded":"2024-03-26T04:11:18.163672+0000","last_fullsized":"2024-03-26T04:11:18.163672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:00:54.996095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'25","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164739+0000","last_change":"2024-03-26T04:10:53.561342+0000","last_active":"2024-03-26T04:11:18.164739+0000","last_peered":"2024-03-26T04:11:18.164739+0000","last_clean":"2024-03-26T04:11:18.164739+0000","last_became_active":"2024-03-26T04:10:53.560905+0000","last_became_peered":"2024-03-26T04:10:53.560905+0000","last_unstale":"2024-03-26T04:11:18.164739+0000","last_undegraded":"2024-03-26T04:11:18.164739+0000","last_fullsized":"2024-03-26T04:11:18.164739+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:16:46.946886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249761+0000","last_change":"2024-03-26T04:10:55.569128+0000","last_active":"2024-03-26T04:11:18.249761+0000","last_peered":"2024-03-26T04:11:18.249761+0000","last_clean":"2024-03-26T04:11:18.249761+0000","last_became_active":"2024-03-26T04:10:55.568630+0000","last_became_peered":"2024-03-26T04:10:55.568630+0000","last_unstale":"2024-03-26T04:11:18.249761+0000","last_undegraded":"2024-03-26T04:11:18.249761+0000","last_fullsized":"2024-03-26T04:11:18.249761+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:27:56.020961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251344+0000","last_change":"2024-03-26T04:10:57.589491+0000","last_active":"2024-03-26T04:11:18.251344+0000","last_peered":"2024-03-26T04:11:18.251344+0000","last_clean":"2024-03-26T04:11:18.251344+0000","last_became_active":"2024-03-26T04:10:57.589153+0000","last_became_peered":"2024-03-26T04:10:57.589153+0000","last_unstale":"2024-03-26T04:11:18.251344+0000","last_undegraded":"2024-03-26T04:11:18.251344+0000","last_fullsized":"2024-03-26T04:11:18.251344+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:46:32.206524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.161845+0000","last_change":"2024-03-26T04:10:53.561369+0000","last_active":"2024-03-26T04:11:18.161845+0000","last_peered":"2024-03-26T04:11:18.161845+0000","last_clean":"2024-03-26T04:11:18.161845+0000","last_became_active":"2024-03-26T04:10:53.560904+0000","last_became_peered":"2024-03-26T04:10:53.560904+0000","last_unstale":"2024-03-26T04:11:18.161845+0000","last_undegraded":"2024-03-26T04:11:18.161845+0000","last_fullsized":"2024-03-26T04:11:18.161845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:31:52.910869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248567+0000","last_change":"2024-03-26T04:10:51.541472+0000","last_active":"2024-03-26T04:11:18.248567+0000","last_peered":"2024-03-26T04:11:18.248567+0000","last_clean":"2024-03-26T04:11:18.248567+0000","last_became_active":"2024-03-26T04:10:51.540938+0000","last_became_peered":"2024-03-26T04:10:51.540938+0000","last_unstale":"2024-03-26T04:11:18.248567+0000","last_undegraded":"2024-03-26T04:11:18.248567+0000","last_fullsized":"2024-03-26T04:11:18.248567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:46:19.914215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251073+0000","last_change":"2024-03-26T04:10:55.570686+0000","last_active":"2024-03-26T04:11:18.251073+0000","last_peered":"2024-03-26T04:11:18.251073+0000","last_clean":"2024-03-26T04:11:18.251073+0000","last_became_active":"2024-03-26T04:10:55.570345+0000","last_became_peered":"2024-03-26T04:10:55.570345+0000","last_unstale":"2024-03-26T04:11:18.251073+0000","last_undegraded":"2024-03-26T04:11:18.251073+0000","last_fullsized":"2024-03-26T04:11:18.251073+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:22:53.826613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162461+0000","last_change":"2024-03-26T04:10:57.585569+0000","last_active":"2024-03-26T04:11:18.162461+0000","last_peered":"2024-03-26T04:11:18.162461+0000","last_clean":"2024-03-26T04:11:18.162461+0000","last_became_active":"2024-03-26T04:10:57.585280+0000","last_became_peered":"2024-03-26T04:10:57.585280+0000","last_unstale":"2024-03-26T04:11:18.162461+0000","last_undegraded":"2024-03-26T04:11:18.162461+0000","last_fullsized":"2024-03-26T04:11:18.162461+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:20:33.540628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249216+0000","last_change":"2024-03-26T04:10:53.559212+0000","last_active":"2024-03-26T04:11:18.249216+0000","last_peered":"2024-03-26T04:11:18.249216+0000","last_clean":"2024-03-26T04:11:18.249216+0000","last_became_active":"2024-03-26T04:10:53.558856+0000","last_became_peered":"2024-03-26T04:10:53.558856+0000","last_unstale":"2024-03-26T04:11:18.249216+0000","last_undegraded":"2024-03-26T04:11:18.249216+0000","last_fullsized":"2024-03-26T04:11:18.249216+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:14:31.138260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164649+0000","last_change":"2024-03-26T04:10:51.545529+0000","last_active":"2024-03-26T04:11:18.164649+0000","last_peered":"2024-03-26T04:11:18.164649+0000","last_clean":"2024-03-26T04:11:18.164649+0000","last_became_active":"2024-03-26T04:10:51.545074+0000","last_became_peered":"2024-03-26T04:10:51.545074+0000","last_unstale":"2024-03-26T04:11:18.164649+0000","last_undegraded":"2024-03-26T04:11:18.164649+0000","last_fullsized":"2024-03-26T04:11:18.164649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:12:14.151922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:20.259642+0000","last_change":"2024-03-26T04:10:37.787106+0000","last_active":"2024-03-26T04:11:20.259642+0000","last_peered":"2024-03-26T04:11:20.259642+0000","last_clean":"2024-03-26T04:11:20.259642+0000","last_became_active":"2024-03-26T04:10:37.786056+0000","last_became_peered":"2024-03-26T04:10:37.786056+0000","last_unstale":"2024-03-26T04:11:20.259642+0000","last_undegraded":"2024-03-26T04:11:20.259642+0000","last_fullsized":"2024-03-26T04:11:20.259642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:36.777647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:36.777647+0000","last_clean_scrub_stamp":"2024-03-26T04:10:36.777647+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:27:47.025092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164459+0000","last_change":"2024-03-26T04:10:55.576476+0000","last_active":"2024-03-26T04:11:18.164459+0000","last_peered":"2024-03-26T04:11:18.164459+0000","last_clean":"2024-03-26T04:11:18.164459+0000","last_became_active":"2024-03-26T04:10:55.576206+0000","last_became_peered":"2024-03-26T04:10:55.576206+0000","last_unstale":"2024-03-26T04:11:18.164459+0000","last_undegraded":"2024-03-26T04:11:18.164459+0000","last_fullsized":"2024-03-26T04:11:18.164459+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:17:16.303231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165298+0000","last_change":"2024-03-26T04:10:57.591736+0000","last_active":"2024-03-26T04:11:18.165298+0000","last_peered":"2024-03-26T04:11:18.165298+0000","last_clean":"2024-03-26T04:11:18.165298+0000","last_became_active":"2024-03-26T04:10:57.590324+0000","last_became_peered":"2024-03-26T04:10:57.590324+0000","last_unstale":"2024-03-26T04:11:18.165298+0000","last_undegraded":"2024-03-26T04:11:18.165298+0000","last_fullsized":"2024-03-26T04:11:18.165298+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:56:35.234213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248804+0000","last_change":"2024-03-26T04:10:53.558715+0000","last_active":"2024-03-26T04:11:18.248804+0000","last_peered":"2024-03-26T04:11:18.248804+0000","last_clean":"2024-03-26T04:11:18.248804+0000","last_became_active":"2024-03-26T04:10:53.557969+0000","last_became_peered":"2024-03-26T04:10:53.557969+0000","last_unstale":"2024-03-26T04:11:18.248804+0000","last_undegraded":"2024-03-26T04:11:18.248804+0000","last_fullsized":"2024-03-26T04:11:18.248804+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:07:26.089390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162751+0000","last_change":"2024-03-26T04:10:51.545819+0000","last_active":"2024-03-26T04:11:18.162751+0000","last_peered":"2024-03-26T04:11:18.162751+0000","last_clean":"2024-03-26T04:11:18.162751+0000","last_became_active":"2024-03-26T04:10:51.545569+0000","last_became_peered":"2024-03-26T04:10:51.545569+0000","last_unstale":"2024-03-26T04:11:18.162751+0000","last_undegraded":"2024-03-26T04:11:18.162751+0000","last_fullsized":"2024-03-26T04:11:18.162751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:51:29.892523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163302+0000","last_change":"2024-03-26T04:10:55.571433+0000","last_active":"2024-03-26T04:11:18.163302+0000","last_peered":"2024-03-26T04:11:18.163302+0000","last_clean":"2024-03-26T04:11:18.163302+0000","last_became_active":"2024-03-26T04:10:55.570925+0000","last_became_peered":"2024-03-26T04:10:55.570925+0000","last_unstale":"2024-03-26T04:11:18.163302+0000","last_undegraded":"2024-03-26T04:11:18.163302+0000","last_fullsized":"2024-03-26T04:11:18.163302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:31:12.320736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163685+0000","last_change":"2024-03-26T04:10:57.589808+0000","last_active":"2024-03-26T04:11:18.163685+0000","last_peered":"2024-03-26T04:11:18.163685+0000","last_clean":"2024-03-26T04:11:18.163685+0000","last_became_active":"2024-03-26T04:10:57.589435+0000","last_became_peered":"2024-03-26T04:10:57.589435+0000","last_unstale":"2024-03-26T04:11:18.163685+0000","last_undegraded":"2024-03-26T04:11:18.163685+0000","last_fullsized":"2024-03-26T04:11:18.163685+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:10:18.894370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165710+0000","last_change":"2024-03-26T04:10:51.542573+0000","last_active":"2024-03-26T04:11:18.165710+0000","last_peered":"2024-03-26T04:11:18.165710+0000","last_clean":"2024-03-26T04:11:18.165710+0000","last_became_active":"2024-03-26T04:10:51.542261+0000","last_became_peered":"2024-03-26T04:10:51.542261+0000","last_unstale":"2024-03-26T04:11:18.165710+0000","last_undegraded":"2024-03-26T04:11:18.165710+0000","last_fullsized":"2024-03-26T04:11:18.165710+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:24:19.907105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'36","reported_seq":94,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:26.589283+0000","last_change":"2024-03-26T04:10:53.558871+0000","last_active":"2024-03-26T04:11:26.589283+0000","last_peered":"2024-03-26T04:11:26.589283+0000","last_clean":"2024-03-26T04:11:26.589283+0000","last_became_active":"2024-03-26T04:10:53.558355+0000","last_became_peered":"2024-03-26T04:10:53.558355+0000","last_unstale":"2024-03-26T04:11:26.589283+0000","last_undegraded":"2024-03-26T04:11:26.589283+0000","last_fullsized":"2024-03-26T04:11:26.589283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:02:11.738180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:26.589744+0000","last_change":"2024-03-26T04:10:55.571065+0000","last_active":"2024-03-26T04:11:26.589744+0000","last_peered":"2024-03-26T04:11:26.589744+0000","last_clean":"2024-03-26T04:11:26.589744+0000","last_became_active":"2024-03-26T04:10:55.570483+0000","last_became_peered":"2024-03-26T04:10:55.570483+0000","last_unstale":"2024-03-26T04:11:26.589744+0000","last_undegraded":"2024-03-26T04:11:26.589744+0000","last_fullsized":"2024-03-26T04:11:26.589744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:41:48.972497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162877+0000","last_change":"2024-03-26T04:10:57.591066+0000","last_active":"2024-03-26T04:11:18.162877+0000","last_peered":"2024-03-26T04:11:18.162877+0000","last_clean":"2024-03-26T04:11:18.162877+0000","last_became_active":"2024-03-26T04:10:57.590376+0000","last_became_peered":"2024-03-26T04:10:57.590376+0000","last_unstale":"2024-03-26T04:11:18.162877+0000","last_undegraded":"2024-03-26T04:11:18.162877+0000","last_fullsized":"2024-03-26T04:11:18.162877+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:32:17.365209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251217+0000","last_change":"2024-03-26T04:10:51.539809+0000","last_active":"2024-03-26T04:11:18.251217+0000","last_peered":"2024-03-26T04:11:18.251217+0000","last_clean":"2024-03-26T04:11:18.251217+0000","last_became_active":"2024-03-26T04:10:51.538894+0000","last_became_peered":"2024-03-26T04:10:51.538894+0000","last_unstale":"2024-03-26T04:11:18.251217+0000","last_undegraded":"2024-03-26T04:11:18.251217+0000","last_fullsized":"2024-03-26T04:11:18.251217+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:31:39.073532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251314+0000","last_change":"2024-03-26T04:10:53.558364+0000","last_active":"2024-03-26T04:11:18.251314+0000","last_peered":"2024-03-26T04:11:18.251314+0000","last_clean":"2024-03-26T04:11:18.251314+0000","last_became_active":"2024-03-26T04:10:53.557947+0000","last_became_peered":"2024-03-26T04:10:53.557947+0000","last_unstale":"2024-03-26T04:11:18.251314+0000","last_undegraded":"2024-03-26T04:11:18.251314+0000","last_fullsized":"2024-03-26T04:11:18.251314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:57:59.390390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:26.589276+0000","last_change":"2024-03-26T04:10:55.572260+0000","last_active":"2024-03-26T04:11:26.589276+0000","last_peered":"2024-03-26T04:11:26.589276+0000","last_clean":"2024-03-26T04:11:26.589276+0000","last_became_active":"2024-03-26T04:10:55.571933+0000","last_became_peered":"2024-03-26T04:10:55.571933+0000","last_unstale":"2024-03-26T04:11:26.589276+0000","last_undegraded":"2024-03-26T04:11:26.589276+0000","last_fullsized":"2024-03-26T04:11:26.589276+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:55:20.499657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162470+0000","last_change":"2024-03-26T04:10:57.585792+0000","last_active":"2024-03-26T04:11:18.162470+0000","last_peered":"2024-03-26T04:11:18.162470+0000","last_clean":"2024-03-26T04:11:18.162470+0000","last_became_active":"2024-03-26T04:10:57.585459+0000","last_became_peered":"2024-03-26T04:10:57.585459+0000","last_unstale":"2024-03-26T04:11:18.162470+0000","last_undegraded":"2024-03-26T04:11:18.162470+0000","last_fullsized":"2024-03-26T04:11:18.162470+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:21:14.735082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165487+0000","last_change":"2024-03-26T04:10:51.539331+0000","last_active":"2024-03-26T04:11:18.165487+0000","last_peered":"2024-03-26T04:11:18.165487+0000","last_clean":"2024-03-26T04:11:18.165487+0000","last_became_active":"2024-03-26T04:10:51.538879+0000","last_became_peered":"2024-03-26T04:10:51.538879+0000","last_unstale":"2024-03-26T04:11:18.165487+0000","last_undegraded":"2024-03-26T04:11:18.165487+0000","last_fullsized":"2024-03-26T04:11:18.165487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:18:04.320518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251143+0000","last_change":"2024-03-26T04:10:53.559627+0000","last_active":"2024-03-26T04:11:18.251143+0000","last_peered":"2024-03-26T04:11:18.251143+0000","last_clean":"2024-03-26T04:11:18.251143+0000","last_became_active":"2024-03-26T04:10:53.559292+0000","last_became_peered":"2024-03-26T04:10:53.559292+0000","last_unstale":"2024-03-26T04:11:18.251143+0000","last_undegraded":"2024-03-26T04:11:18.251143+0000","last_fullsized":"2024-03-26T04:11:18.251143+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:26:16.142401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162113+0000","last_change":"2024-03-26T04:10:55.572375+0000","last_active":"2024-03-26T04:11:18.162113+0000","last_peered":"2024-03-26T04:11:18.162113+0000","last_clean":"2024-03-26T04:11:18.162113+0000","last_became_active":"2024-03-26T04:10:55.570814+0000","last_became_peered":"2024-03-26T04:10:55.570814+0000","last_unstale":"2024-03-26T04:11:18.162113+0000","last_undegraded":"2024-03-26T04:11:18.162113+0000","last_fullsized":"2024-03-26T04:11:18.162113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:24:39.056159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164578+0000","last_change":"2024-03-26T04:10:57.591329+0000","last_active":"2024-03-26T04:11:18.164578+0000","last_peered":"2024-03-26T04:11:18.164578+0000","last_clean":"2024-03-26T04:11:18.164578+0000","last_became_active":"2024-03-26T04:10:57.590320+0000","last_became_peered":"2024-03-26T04:10:57.590320+0000","last_unstale":"2024-03-26T04:11:18.164578+0000","last_undegraded":"2024-03-26T04:11:18.164578+0000","last_fullsized":"2024-03-26T04:11:18.164578+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:35:02.778257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:26.589571+0000","last_change":"2024-03-26T04:10:55.575391+0000","last_active":"2024-03-26T04:11:26.589571+0000","last_peered":"2024-03-26T04:11:26.589571+0000","last_clean":"2024-03-26T04:11:26.589571+0000","last_became_active":"2024-03-26T04:10:55.574403+0000","last_became_peered":"2024-03-26T04:10:55.574403+0000","last_unstale":"2024-03-26T04:11:26.589571+0000","last_undegraded":"2024-03-26T04:11:26.589571+0000","last_fullsized":"2024-03-26T04:11:26.589571+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:31:27.389111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162614+0000","last_change":"2024-03-26T04:10:51.545944+0000","last_active":"2024-03-26T04:11:18.162614+0000","last_peered":"2024-03-26T04:11:18.162614+0000","last_clean":"2024-03-26T04:11:18.162614+0000","last_became_active":"2024-03-26T04:10:51.545625+0000","last_became_peered":"2024-03-26T04:10:51.545625+0000","last_unstale":"2024-03-26T04:11:18.162614+0000","last_undegraded":"2024-03-26T04:11:18.162614+0000","last_fullsized":"2024-03-26T04:11:18.162614+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:24:04.916197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162468+0000","last_change":"2024-03-26T04:10:53.561227+0000","last_active":"2024-03-26T04:11:18.162468+0000","last_peered":"2024-03-26T04:11:18.162468+0000","last_clean":"2024-03-26T04:11:18.162468+0000","last_became_active":"2024-03-26T04:10:53.560743+0000","last_became_peered":"2024-03-26T04:10:53.560743+0000","last_unstale":"2024-03-26T04:11:18.162468+0000","last_undegraded":"2024-03-26T04:11:18.162468+0000","last_fullsized":"2024-03-26T04:11:18.162468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:24:19.721900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163541+0000","last_change":"2024-03-26T04:10:57.586333+0000","last_active":"2024-03-26T04:11:18.163541+0000","last_peered":"2024-03-26T04:11:18.163541+0000","last_clean":"2024-03-26T04:11:18.163541+0000","last_became_active":"2024-03-26T04:10:57.586097+0000","last_became_peered":"2024-03-26T04:10:57.586097+0000","last_unstale":"2024-03-26T04:11:18.163541+0000","last_undegraded":"2024-03-26T04:11:18.163541+0000","last_fullsized":"2024-03-26T04:11:18.163541+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:57:39.245475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.250620+0000","last_change":"2024-03-26T04:10:55.571111+0000","last_active":"2024-03-26T04:11:18.250620+0000","last_peered":"2024-03-26T04:11:18.250620+0000","last_clean":"2024-03-26T04:11:18.250620+0000","last_became_active":"2024-03-26T04:10:55.570718+0000","last_became_peered":"2024-03-26T04:10:55.570718+0000","last_unstale":"2024-03-26T04:11:18.250620+0000","last_undegraded":"2024-03-26T04:11:18.250620+0000","last_fullsized":"2024-03-26T04:11:18.250620+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:36:31.538712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248495+0000","last_change":"2024-03-26T04:10:51.542052+0000","last_active":"2024-03-26T04:11:18.248495+0000","last_peered":"2024-03-26T04:11:18.248495+0000","last_clean":"2024-03-26T04:11:18.248495+0000","last_became_active":"2024-03-26T04:10:51.540974+0000","last_became_peered":"2024-03-26T04:10:51.540974+0000","last_unstale":"2024-03-26T04:11:18.248495+0000","last_undegraded":"2024-03-26T04:11:18.248495+0000","last_fullsized":"2024-03-26T04:11:18.248495+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:51:47.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162232+0000","last_change":"2024-03-26T04:10:53.561000+0000","last_active":"2024-03-26T04:11:18.162232+0000","last_peered":"2024-03-26T04:11:18.162232+0000","last_clean":"2024-03-26T04:11:18.162232+0000","last_became_active":"2024-03-26T04:10:53.560703+0000","last_became_peered":"2024-03-26T04:10:53.560703+0000","last_unstale":"2024-03-26T04:11:18.162232+0000","last_undegraded":"2024-03-26T04:11:18.162232+0000","last_fullsized":"2024-03-26T04:11:18.162232+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:47:03.620113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165093+0000","last_change":"2024-03-26T04:10:57.585042+0000","last_active":"2024-03-26T04:11:18.165093+0000","last_peered":"2024-03-26T04:11:18.165093+0000","last_clean":"2024-03-26T04:11:18.165093+0000","last_became_active":"2024-03-26T04:10:57.584229+0000","last_became_peered":"2024-03-26T04:10:57.584229+0000","last_unstale":"2024-03-26T04:11:18.165093+0000","last_undegraded":"2024-03-26T04:11:18.165093+0000","last_fullsized":"2024-03-26T04:11:18.165093+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:01:21.799032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163317+0000","last_change":"2024-03-26T04:10:51.539367+0000","last_active":"2024-03-26T04:11:18.163317+0000","last_peered":"2024-03-26T04:11:18.163317+0000","last_clean":"2024-03-26T04:11:18.163317+0000","last_became_active":"2024-03-26T04:10:51.539050+0000","last_became_peered":"2024-03-26T04:10:51.539050+0000","last_unstale":"2024-03-26T04:11:18.163317+0000","last_undegraded":"2024-03-26T04:11:18.163317+0000","last_fullsized":"2024-03-26T04:11:18.163317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:30:52.553832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249964+0000","last_change":"2024-03-26T04:10:53.559236+0000","last_active":"2024-03-26T04:11:18.249964+0000","last_peered":"2024-03-26T04:11:18.249964+0000","last_clean":"2024-03-26T04:11:18.249964+0000","last_became_active":"2024-03-26T04:10:53.558749+0000","last_became_peered":"2024-03-26T04:10:53.558749+0000","last_unstale":"2024-03-26T04:11:18.249964+0000","last_undegraded":"2024-03-26T04:11:18.249964+0000","last_fullsized":"2024-03-26T04:11:18.249964+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:53:29.311641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163896+0000","last_change":"2024-03-26T04:10:55.575639+0000","last_active":"2024-03-26T04:11:18.163896+0000","last_peered":"2024-03-26T04:11:18.163896+0000","last_clean":"2024-03-26T04:11:18.163896+0000","last_became_active":"2024-03-26T04:10:55.574637+0000","last_became_peered":"2024-03-26T04:10:55.574637+0000","last_unstale":"2024-03-26T04:11:18.163896+0000","last_undegraded":"2024-03-26T04:11:18.163896+0000","last_fullsized":"2024-03-26T04:11:18.163896+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:43:55.496812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249385+0000","last_change":"2024-03-26T04:10:57.589110+0000","last_active":"2024-03-26T04:11:18.249385+0000","last_peered":"2024-03-26T04:11:18.249385+0000","last_clean":"2024-03-26T04:11:18.249385+0000","last_became_active":"2024-03-26T04:10:57.588387+0000","last_became_peered":"2024-03-26T04:10:57.588387+0000","last_unstale":"2024-03-26T04:11:18.249385+0000","last_undegraded":"2024-03-26T04:11:18.249385+0000","last_fullsized":"2024-03-26T04:11:18.249385+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:20:59.791679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:26.589655+0000","last_change":"2024-03-26T04:10:55.575876+0000","last_active":"2024-03-26T04:11:26.589655+0000","last_peered":"2024-03-26T04:11:26.589655+0000","last_clean":"2024-03-26T04:11:26.589655+0000","last_became_active":"2024-03-26T04:10:55.575509+0000","last_became_peered":"2024-03-26T04:10:55.575509+0000","last_unstale":"2024-03-26T04:11:26.589655+0000","last_undegraded":"2024-03-26T04:11:26.589655+0000","last_fullsized":"2024-03-26T04:11:26.589655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:08:53.905936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163542+0000","last_change":"2024-03-26T04:10:51.539899+0000","last_active":"2024-03-26T04:11:18.163542+0000","last_peered":"2024-03-26T04:11:18.163542+0000","last_clean":"2024-03-26T04:11:18.163542+0000","last_became_active":"2024-03-26T04:10:51.539497+0000","last_became_peered":"2024-03-26T04:10:51.539497+0000","last_unstale":"2024-03-26T04:11:18.163542+0000","last_undegraded":"2024-03-26T04:11:18.163542+0000","last_fullsized":"2024-03-26T04:11:18.163542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:09:06.666206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'25","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163412+0000","last_change":"2024-03-26T04:10:53.562993+0000","last_active":"2024-03-26T04:11:18.163412+0000","last_peered":"2024-03-26T04:11:18.163412+0000","last_clean":"2024-03-26T04:11:18.163412+0000","last_became_active":"2024-03-26T04:10:53.561927+0000","last_became_peered":"2024-03-26T04:10:53.561927+0000","last_unstale":"2024-03-26T04:11:18.163412+0000","last_undegraded":"2024-03-26T04:11:18.163412+0000","last_fullsized":"2024-03-26T04:11:18.163412+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:58:41.744418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162674+0000","last_change":"2024-03-26T04:10:57.584793+0000","last_active":"2024-03-26T04:11:18.162674+0000","last_peered":"2024-03-26T04:11:18.162674+0000","last_clean":"2024-03-26T04:11:18.162674+0000","last_became_active":"2024-03-26T04:10:57.584509+0000","last_became_peered":"2024-03-26T04:10:57.584509+0000","last_unstale":"2024-03-26T04:11:18.162674+0000","last_undegraded":"2024-03-26T04:11:18.162674+0000","last_fullsized":"2024-03-26T04:11:18.162674+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:55:26.088975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163248+0000","last_change":"2024-03-26T04:10:51.540059+0000","last_active":"2024-03-26T04:11:18.163248+0000","last_peered":"2024-03-26T04:11:18.163248+0000","last_clean":"2024-03-26T04:11:18.163248+0000","last_became_active":"2024-03-26T04:10:51.539604+0000","last_became_peered":"2024-03-26T04:10:51.539604+0000","last_unstale":"2024-03-26T04:11:18.163248+0000","last_undegraded":"2024-03-26T04:11:18.163248+0000","last_fullsized":"2024-03-26T04:11:18.163248+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:48:36.654899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164751+0000","last_change":"2024-03-26T04:10:53.560915+0000","last_active":"2024-03-26T04:11:18.164751+0000","last_peered":"2024-03-26T04:11:18.164751+0000","last_clean":"2024-03-26T04:11:18.164751+0000","last_became_active":"2024-03-26T04:10:53.560168+0000","last_became_peered":"2024-03-26T04:10:53.560168+0000","last_unstale":"2024-03-26T04:11:18.164751+0000","last_undegraded":"2024-03-26T04:11:18.164751+0000","last_fullsized":"2024-03-26T04:11:18.164751+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:57:11.873572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.163206+0000","last_change":"2024-03-26T04:10:55.576228+0000","last_active":"2024-03-26T04:11:18.163206+0000","last_peered":"2024-03-26T04:11:18.163206+0000","last_clean":"2024-03-26T04:11:18.163206+0000","last_became_active":"2024-03-26T04:10:55.575736+0000","last_became_peered":"2024-03-26T04:10:55.575736+0000","last_unstale":"2024-03-26T04:11:18.163206+0000","last_undegraded":"2024-03-26T04:11:18.163206+0000","last_fullsized":"2024-03-26T04:11:18.163206+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:08:41.061088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.248221+0000","last_change":"2024-03-26T04:10:57.589354+0000","last_active":"2024-03-26T04:11:18.248221+0000","last_peered":"2024-03-26T04:11:18.248221+0000","last_clean":"2024-03-26T04:11:18.248221+0000","last_became_active":"2024-03-26T04:10:57.588529+0000","last_became_peered":"2024-03-26T04:10:57.588529+0000","last_unstale":"2024-03-26T04:11:18.248221+0000","last_undegraded":"2024-03-26T04:11:18.248221+0000","last_fullsized":"2024-03-26T04:11:18.248221+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:07:23.322125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165440+0000","last_change":"2024-03-26T04:10:51.540146+0000","last_active":"2024-03-26T04:11:18.165440+0000","last_peered":"2024-03-26T04:11:18.165440+0000","last_clean":"2024-03-26T04:11:18.165440+0000","last_became_active":"2024-03-26T04:10:51.539833+0000","last_became_peered":"2024-03-26T04:10:51.539833+0000","last_unstale":"2024-03-26T04:11:18.165440+0000","last_undegraded":"2024-03-26T04:11:18.165440+0000","last_fullsized":"2024-03-26T04:11:18.165440+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:24:27.959472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.164234+0000","last_change":"2024-03-26T04:10:53.561376+0000","last_active":"2024-03-26T04:11:18.164234+0000","last_peered":"2024-03-26T04:11:18.164234+0000","last_clean":"2024-03-26T04:11:18.164234+0000","last_became_active":"2024-03-26T04:10:53.560948+0000","last_became_peered":"2024-03-26T04:10:53.560948+0000","last_unstale":"2024-03-26T04:11:18.164234+0000","last_undegraded":"2024-03-26T04:11:18.164234+0000","last_fullsized":"2024-03-26T04:11:18.164234+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:31:33.079418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.249363+0000","last_change":"2024-03-26T04:10:55.570508+0000","last_active":"2024-03-26T04:11:18.249363+0000","last_peered":"2024-03-26T04:11:18.249363+0000","last_clean":"2024-03-26T04:11:18.249363+0000","last_became_active":"2024-03-26T04:10:55.570063+0000","last_became_peered":"2024-03-26T04:10:55.570063+0000","last_unstale":"2024-03-26T04:11:18.249363+0000","last_undegraded":"2024-03-26T04:11:18.249363+0000","last_fullsized":"2024-03-26T04:11:18.249363+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:25:02.612299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162748+0000","last_change":"2024-03-26T04:10:57.591470+0000","last_active":"2024-03-26T04:11:18.162748+0000","last_peered":"2024-03-26T04:11:18.162748+0000","last_clean":"2024-03-26T04:11:18.162748+0000","last_became_active":"2024-03-26T04:10:57.591077+0000","last_became_peered":"2024-03-26T04:10:57.591077+0000","last_unstale":"2024-03-26T04:11:18.162748+0000","last_undegraded":"2024-03-26T04:11:18.162748+0000","last_fullsized":"2024-03-26T04:11:18.162748+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:56.566925+0000","last_clean_scrub_stamp":"2024-03-26T04:10:56.566925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:42:40.920656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.251052+0000","last_change":"2024-03-26T04:10:51.541457+0000","last_active":"2024-03-26T04:11:18.251052+0000","last_peered":"2024-03-26T04:11:18.251052+0000","last_clean":"2024-03-26T04:11:18.251052+0000","last_became_active":"2024-03-26T04:10:51.540871+0000","last_became_peered":"2024-03-26T04:10:51.540871+0000","last_unstale":"2024-03-26T04:11:18.251052+0000","last_undegraded":"2024-03-26T04:11:18.251052+0000","last_fullsized":"2024-03-26T04:11:18.251052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:50.519979+0000","last_clean_scrub_stamp":"2024-03-26T04:10:50.519979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:51:16.523756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.165587+0000","last_change":"2024-03-26T04:10:53.563631+0000","last_active":"2024-03-26T04:11:18.165587+0000","last_peered":"2024-03-26T04:11:18.165587+0000","last_clean":"2024-03-26T04:11:18.165587+0000","last_became_active":"2024-03-26T04:10:53.562681+0000","last_became_peered":"2024-03-26T04:10:53.562681+0000","last_unstale":"2024-03-26T04:11:18.165587+0000","last_undegraded":"2024-03-26T04:11:18.165587+0000","last_fullsized":"2024-03-26T04:11:18.165587+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:52.535394+0000","last_clean_scrub_stamp":"2024-03-26T04:10:52.535394+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:47:16.349668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T04:11:18.162372+0000","last_change":"2024-03-26T04:10:55.570942+0000","last_active":"2024-03-26T04:11:18.162372+0000","last_peered":"2024-03-26T04:11:18.162372+0000","last_clean":"2024-03-26T04:11:18.162372+0000","last_became_active":"2024-03-26T04:10:55.570516+0000","last_became_peered":"2024-03-26T04:10:55.570516+0000","last_unstale":"2024-03-26T04:11:18.162372+0000","last_undegraded":"2024-03-26T04:11:18.162372+0000","last_fullsized":"2024-03-26T04:11:18.162372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:54.549291+0000","last_clean_scrub_stamp":"2024-03-26T04:10:54.549291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:10:29.587494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411343,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738392,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:11:39.047 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T04:11:39.047 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T04:11:39.047 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T04:11:39.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T04:11:39.275 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:39 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ::ffff:172.21.15.149 - - [26/Mar/2024:04:11:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:39.276 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:39 smithi136 ceph-mon[25848]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:11:39.487 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:39 smithi149 ceph-mon[30657]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:11:39.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:39 smithi204 ceph-mon[30652]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:11:40.430 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:40 smithi136 ceph-mon[25848]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:40.487 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:40 smithi149 ceph-mon[30657]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:40.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:40 smithi204 ceph-mon[30652]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:41.084 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:41.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:41 smithi136 ceph-mon[25848]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:41.487 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:41 smithi149 ceph-mon[30657]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:41.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:41 smithi204 ceph-mon[30652]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:43.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:43 smithi136 ceph-mon[25848]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:43.449 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 podman[57838]: 2024-03-26 04:11:43.378293438 +0000 UTC m=+0.403860146 container create 21621393e316a6b6ef7cf27cc564c63ee86484d75e53c2ae9dcb70a62f774ae8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:11:43.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:43 smithi149 ceph-mon[30657]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:43.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:43 smithi204 ceph-mon[30652]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:44.023 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 podman[57838]: 2024-03-26 04:11:43.915153305 +0000 UTC m=+0.940720017 container init 21621393e316a6b6ef7cf27cc564c63ee86484d75e53c2ae9dcb70a62f774ae8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:11:44.023 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 podman[57838]: 2024-03-26 04:11:43.920478918 +0000 UTC m=+0.946045627 container start 21621393e316a6b6ef7cf27cc564c63ee86484d75e53c2ae9dcb70a62f774ae8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:11:44.024 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:11:43.932Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T04:11:44.024 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:11:43.932Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T04:11:44.024 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:11:43.933Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.136 port=9094 2024-03-26T04:11:44.024 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:11:43.934Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T04:11:44.024 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:11:43.957Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:11:44.024 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:11:43.957Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:11:44.024 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:11:43.960Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T04:11:44.024 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:43 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:11:43.960Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T04:11:44.024 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:44 smithi136 bash[57838]: 21621393e316a6b6ef7cf27cc564c63ee86484d75e53c2ae9dcb70a62f774ae8 2024-03-26T04:11:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T04:11:44.091 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T04:11:44.312 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:44 smithi136 systemd[1]: Started Ceph alertmanager.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:11:44.312 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:11:44] ENGINE Bus STOPPING 2024-03-26T04:11:44.312 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:11:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:11:44.312 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:11:44] ENGINE Bus STOPPED 2024-03-26T04:11:44.312 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:11:44] ENGINE Bus STARTING 2024-03-26T04:11:44.312 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:11:44] ENGINE Serving on http://:::9283 2024-03-26T04:11:44.660 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:44 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:11:44] ENGINE Bus STARTED 2024-03-26T04:11:44.687 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T04:11:44.687 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T04:11:44.687 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T04:11:44.697 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi136.front.sepia.ceph.com 2024-03-26T04:11:44.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-26T04:11:44.932 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:45.363 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:45 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:45 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:45 smithi136 ceph-mon[25848]: from='client.? 172.21.15.136:0/205608710' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:11:45.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:45 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:45 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:45 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:45 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:45 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:11:45.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:45 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:45 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:45 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:45 smithi149 ceph-mon[30657]: from='client.? 172.21.15.136:0/205608710' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:11:45.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:45 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:45 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:45 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:45 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:45 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:11:45.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:45 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:45 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:45 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:45 smithi204 ceph-mon[30652]: from='client.? 172.21.15.136:0/205608710' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:11:45.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:45 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:45 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:45 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:45 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:45.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:45 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:11:45.498 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:45 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:46.077 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:45 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:11:45.934Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000711707s 2024-03-26T04:11:46.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:46 smithi136 ceph-mon[25848]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:11:46.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:46 smithi136 ceph-mon[25848]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:46.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:46 smithi136 ceph-mon[25848]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:11:46.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:46 smithi136 ceph-mon[25848]: Deploying daemon grafana.a on smithi149 2024-03-26T04:11:46.398 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:46 smithi204 ceph-mon[30652]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:11:46.399 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:46 smithi204 ceph-mon[30652]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:46.399 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:46 smithi204 ceph-mon[30652]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:11:46.399 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:46 smithi204 ceph-mon[30652]: Deploying daemon grafana.a on smithi149 2024-03-26T04:11:46.487 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:46 smithi149 ceph-mon[30657]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:11:46.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:46 smithi149 ceph-mon[30657]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:46.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:46 smithi149 ceph-mon[30657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:11:46.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:46 smithi149 ceph-mon[30657]: Deploying daemon grafana.a on smithi149 2024-03-26T04:11:46.848 INFO:teuthology.orchestra.run.smithi136.stdout:167 167 2024-03-26T04:11:47.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:47 smithi136 ceph-mon[25848]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:47.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-26T04:11:47.487 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:47 smithi149 ceph-mon[30657]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:47.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:47 smithi204 ceph-mon[30652]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:47.722 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:49.201 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:49 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ::ffff:172.21.15.149 - - [26/Mar/2024:04:11:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:49.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:49 smithi204 ceph-mon[30652]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:49.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:49 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:49.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:49 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:49.521 INFO:teuthology.orchestra.run.smithi136.stdout:Backend: cephadm 2024-03-26T04:11:49.522 INFO:teuthology.orchestra.run.smithi136.stdout:Available: Yes 2024-03-26T04:11:49.522 INFO:teuthology.orchestra.run.smithi136.stdout:Paused: No 2024-03-26T04:11:49.522 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:49 smithi136 ceph-mon[25848]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:49.522 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:49 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:49.522 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:49 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:49.596 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:49 smithi149 ceph-mon[30657]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:49.596 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:49 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:49.596 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:49 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:50.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-26T04:11:50.351 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:50.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:50 smithi136 ceph-mon[25848]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:50.676 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:50 smithi149 ceph-mon[30657]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:50.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:50 smithi204 ceph-mon[30652]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:51.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:51 smithi136 ceph-mon[25848]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:51.737 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:51 smithi149 ceph-mon[30657]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:51.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:51 smithi204 ceph-mon[30652]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:52.150 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:11:52.150 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 starting - - - - 2024-03-26T04:11:52.150 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi149 *:8443,8765 running (2m) 31s ago 2m 448M - 19.0.0-1775-g5e70124b 5f21c9264136 f6579a30c564 2024-03-26T04:11:52.150 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:9283,8765 running (4m) 24s ago 4m 483M - 19.0.0-1775-g5e70124b 5f21c9264136 06ee71789a25 2024-03-26T04:11:52.151 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.z smithi204 *:8443,8765 running (2m) 32s ago 2m 447M - 19.0.0-1775-g5e70124b 5f21c9264136 857705d92f93 2024-03-26T04:11:52.151 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (4m) 24s ago 4m 53.7M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 b6196b95c1e4 2024-03-26T04:11:52.151 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi149 running (2m) 31s ago 2m 42.7M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a2d968c1e135 2024-03-26T04:11:52.151 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi204 running (2m) 32s ago 2m 45.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fce8ab718fac 2024-03-26T04:11:52.151 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (2m) 24s ago 2m 61.9M 15.6G 19.0.0-1775-g5e70124b 5f21c9264136 cccd09bd51a8 2024-03-26T04:11:52.151 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi149 running (107s) 31s ago 107s 52.3M 4792M 19.0.0-1775-g5e70124b 5f21c9264136 a52de1a6605b 2024-03-26T04:11:52.151 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi204 running (83s) 32s ago 83s 62.5M 16.6G 19.0.0-1775-g5e70124b 5f21c9264136 18f0a4b504d4 2024-03-26T04:11:52.151 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi149 *:9095 running (42s) 31s ago 42s 26.4M - 2.43.0 a07b618ecd1d 669c70d35005 2024-03-26T04:11:52.151 INFO:teuthology.orchestra.run.smithi136.stdout:rgw.foo.a smithi136 *:80 running (62s) 24s ago 62s 79.9M - 19.0.0-1775-g5e70124b 5f21c9264136 d4d28675bacd 2024-03-26T04:11:52.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-26T04:11:53.107 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:53.473 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:53 smithi136 ceph-mon[25848]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:53.473 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:53 smithi136 ceph-mon[25848]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:53.737 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:53 smithi149 ceph-mon[30657]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:53.737 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:53 smithi149 ceph-mon[30657]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:53.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:53 smithi204 ceph-mon[30652]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:53.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:53 smithi204 ceph-mon[30652]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:54.198 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:11:53 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:11:53.937Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003587573s 2024-03-26T04:11:54.995 INFO:teuthology.orchestra.run.smithi136.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-26T04:11:54.995 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager ?:9093,9094 0/1 - 60s smithi136=a;count:1 2024-03-26T04:11:54.995 INFO:teuthology.orchestra.run.smithi136.stdout:grafana ?:3000 0/1 - 57s smithi149=a;count:1 2024-03-26T04:11:54.995 INFO:teuthology.orchestra.run.smithi136.stdout:mgr 3/3 35s ago 2m smithi136=y;smithi149=x;smithi204=z;count:3 2024-03-26T04:11:54.996 INFO:teuthology.orchestra.run.smithi136.stdout:mon 3/3 35s ago 3m smithi136:172.21.15.136=a;smithi149:172.21.15.149=b;smithi204:172.21.15.204=c;count:3 2024-03-26T04:11:54.996 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter ?:9100 0/3 - 63s smithi136=a;smithi149=b;smithi204=c;count:3 2024-03-26T04:11:54.996 INFO:teuthology.orchestra.run.smithi136.stdout:osd 3 35s ago - 2024-03-26T04:11:54.996 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus ?:9095 1/1 34s ago 65s smithi149=a;count:1 2024-03-26T04:11:54.996 INFO:teuthology.orchestra.run.smithi136.stdout:rgw.foo.a ?:80 1/1 27s ago 64s smithi136=foo.a;count:1 2024-03-26T04:11:55.387 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:55 smithi149 ceph-mon[30657]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:55.387 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:55 smithi149 ceph-mon[30657]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:55.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:55 smithi136 ceph-mon[25848]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:55.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:55 smithi136 ceph-mon[25848]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:55.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-26T04:11:55.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:55 smithi204 ceph-mon[30652]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:55.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:55 smithi204 ceph-mon[30652]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:55.966 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:57.238 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 systemd[1]: Starting Ceph grafana.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:11:57.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:57 smithi136 ceph-mon[25848]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:57.497 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:57 smithi204 ceph-mon[30652]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:57.581 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-mon[30657]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:11:57.582 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 podman[42966]: 2024-03-26 04:11:57.239835345 +0000 UTC m=+0.017059877 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-26T04:11:57.582 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 podman[42966]: 2024-03-26 04:11:57.35471937 +0000 UTC m=+0.131943945 container create e6ff275b36d8df7384c111b257500a0dd8dc2f4215378fd7c0b4ccb082dbbd63 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a, distribution-scope=public, description=Ceph Grafana Container, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, version=8.5, release=239.1651231664, io.openshift.expose-services=, vcs-type=git, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, name=ubi8, maintainer=Paul Cuzner , vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-26T04:11:57.582 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 podman[42966]: 2024-03-26 04:11:57.581991381 +0000 UTC m=+0.359215918 container init e6ff275b36d8df7384c111b257500a0dd8dc2f4215378fd7c0b4ccb082dbbd63 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a, version=8.5, maintainer=Paul Cuzner , distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, architecture=x86_64, name=ubi8, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-26T04:11:57.832 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 podman[42966]: 2024-03-26 04:11:57.587459283 +0000 UTC m=+0.364683816 container start e6ff275b36d8df7384c111b257500a0dd8dc2f4215378fd7c0b4ccb082dbbd63 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, io.buildah.version=1.32.0, name=ubi8, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 bash[42966]: e6ff275b36d8df7384c111b257500a0dd8dc2f4215378fd7c0b4ccb082dbbd63 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 systemd[1]: Started Ceph grafana.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742498191Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742737329Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742756761Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742763753Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742769962Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742776459Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.74278228Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742788351Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T04:11:57.833 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742794451Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T04:11:57.834 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742800436Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T04:11:57.834 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742805976Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T04:11:57.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742811539Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T04:11:57.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.742817376Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T04:11:57.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=settings t=2024-03-26T04:11:57.74283888Z level=info msg="App mode production" 2024-03-26T04:11:57.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=sqlstore t=2024-03-26T04:11:57.742885593Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T04:11:57.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=sqlstore t=2024-03-26T04:11:57.742902077Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T04:11:57.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.743793802Z level=info msg="Starting DB migrations" 2024-03-26T04:11:57.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.743872071Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T04:11:57.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.744398355Z level=info msg="Executing migration" id="create user table" 2024-03-26T04:11:57.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.744863912Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T04:11:57.835 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.745488818Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.745955431Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.746380208Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.746790382Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.747507449Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.747934004Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.748317421Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.749205628Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.749571514Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.749931912Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.750319365Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T04:11:57.836 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.750589368Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.751059804Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.751393669Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.751822136Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.75221856Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.752608392Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.753822311Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.754255383Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.754715451Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.755134707Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.756084218Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.756555502Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.757495804Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.758272296Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T04:11:57.837 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.758717618Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.759695242Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.760411353Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.761300987Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.761740499Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.762719049Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.763468969Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.764309284Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.765175449Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.766042013Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.766895052Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.767752784Z level=info msg="Executing migration" id="create star table" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.768603726Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T04:11:57.838 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.769666554Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.770349897Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.77079493Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.774420491Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.777038925Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.77747895Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.780079501Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.78291179Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.78574384Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.788598913Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.791361383Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.794258329Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T04:11:57.839 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.796847848Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T04:11:57.840 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.799774992Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T04:11:57.840 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.802628541Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T04:11:57.840 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.805357466Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T04:11:57.840 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.808182596Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T04:11:57.840 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.810925988Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T04:11:57.840 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.813689748Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T04:11:57.840 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.816647786Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T04:11:57.840 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.819560795Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T04:11:57.841 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.822227224Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T04:11:57.841 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.824848824Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T04:11:57.841 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.827470838Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T04:11:57.841 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.830028005Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T04:11:57.841 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.832818045Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T04:11:57.841 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.83534287Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T04:11:57.912 INFO:teuthology.orchestra.run.smithi136.stdout:HOST ADDR LABELS STATUS 2024-03-26T04:11:57.912 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 172.21.15.136 2024-03-26T04:11:57.912 INFO:teuthology.orchestra.run.smithi136.stdout:smithi149 172.21.15.149 2024-03-26T04:11:57.913 INFO:teuthology.orchestra.run.smithi136.stdout:smithi204 172.21.15.204 2024-03-26T04:11:57.913 INFO:teuthology.orchestra.run.smithi136.stdout:3 hosts in cluster 2024-03-26T04:11:58.085 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.837821628Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T04:11:58.085 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.840694065Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T04:11:58.085 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.842985637Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T04:11:58.085 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.84331938Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T04:11:58.085 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.844260684Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T04:11:58.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.844842942Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T04:11:58.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.845389056Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T04:11:58.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.846246742Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T04:11:58.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.847231461Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T04:11:58.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.848016772Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T04:11:58.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.84886317Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T04:11:58.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.849726049Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T04:11:58.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.850611729Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T04:11:58.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.851624434Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T04:11:58.086 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.854374677Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T04:11:58.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.857282214Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T04:11:58.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.860088008Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T04:11:58.087 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.862846119Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T04:11:58.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.865441201Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T04:11:58.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.868046219Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T04:11:58.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.871013881Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T04:11:58.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.873605135Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T04:11:58.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.876318089Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T04:11:58.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.879201259Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T04:11:58.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.88220545Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T04:11:58.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.884927922Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T04:11:58.088 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.887682773Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T04:11:58.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.89041692Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T04:11:58.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.893362973Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T04:11:58.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.895990159Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T04:11:58.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.899116643Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T04:11:58.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.899676166Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T04:11:58.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.902402438Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T04:11:58.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.90487703Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T04:11:58.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.907506818Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T04:11:58.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.908048844Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T04:11:58.089 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.910612217Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.911192063Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.913420904Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.915793009Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.916147311Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.91881277Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.921463217Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.924106194Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.924850337Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.927541079Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.930149285Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.932931064Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.933383503Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T04:11:58.090 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.934229954Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.93507228Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.935858371Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.936275067Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.937251455Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.93799887Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.939122798Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.939950182Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.940772105Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.941789006Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.942404343Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.943254687Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.944098693Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.944916928Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T04:11:58.091 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.945773234Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.946614257Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.947603167Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.948511176Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.949253523Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.95009276Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.95099Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.951963867Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.952672467Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.953673302Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.954391929Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.955214321Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.956121909Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T04:11:58.092 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.956993464Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T04:11:58.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.957985127Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T04:11:58.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.958755814Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T04:11:58.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.959757713Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T04:11:58.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.960702706Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T04:11:58.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.961350736Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T04:11:58.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.962337931Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T04:11:58.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.963087505Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T04:11:58.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.96579807Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T04:11:58.093 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.968634475Z level=info msg="Executing migration" id="create session table" 2024-03-26T04:11:58.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.971362074Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T04:11:58.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.974343232Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T04:11:58.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.977254159Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T04:11:58.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.979990105Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T04:11:58.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.982896847Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T04:11:58.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.985704906Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T04:11:58.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.988410858Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T04:11:58.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.991311192Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T04:11:58.094 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.994267567Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.997147587Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:57 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:57.999884331Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.002477191Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.00540201Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.008330594Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.011314316Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.013786897Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.016772639Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.019216942Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.021663713Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.023935724Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T04:11:58.095 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.026702834Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.02922508Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.031776063Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.034591505Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.037114007Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.039787864Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.042270548Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.044889207Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.047655595Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.050081559Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.052665139Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.055256824Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.057941Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.058660337Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.059651628Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T04:11:58.096 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.060366958Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T04:11:58.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.061294334Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T04:11:58.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.062036671Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T04:11:58.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.06288964Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T04:11:58.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.063855675Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T04:11:58.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.064566611Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T04:11:58.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.065410817Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T04:11:58.097 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.066197519Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T04:11:58.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.067050515Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T04:11:58.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.06789403Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T04:11:58.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.068728518Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T04:11:58.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.071297578Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T04:11:58.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.074059629Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T04:11:58.098 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.076688919Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T04:11:58.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.079558085Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T04:11:58.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.082388296Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T04:11:58.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.085338231Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T04:11:58.099 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.087900546Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T04:11:58.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.090592931Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T04:11:58.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.093321044Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T04:11:58.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.096025341Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T04:11:58.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.098796678Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T04:11:58.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.101295691Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T04:11:58.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.103644729Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T04:11:58.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.106166122Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T04:11:58.339 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.108896654Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T04:11:58.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.111577973Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T04:11:58.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.114347767Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T04:11:58.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.116969097Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T04:11:58.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.119715622Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T04:11:58.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.122267055Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T04:11:58.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.125003489Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T04:11:58.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.127820616Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T04:11:58.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.130381423Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T04:11:58.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.131141226Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T04:11:58.340 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.131913804Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T04:11:58.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.132712798Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T04:11:58.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.133580801Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T04:11:58.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.134511489Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T04:11:58.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.135273014Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T04:11:58.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.13621699Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T04:11:58.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.136949244Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T04:11:58.341 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.137774081Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.138628578Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.139437749Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.140431771Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.141166889Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.14207425Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.142622025Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.143065748Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.143902328Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.144357329Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.14480004Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.145632465Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T04:11:58.342 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.146051288Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.146926478Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.147794978Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.148642687Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.149592304Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.150338133Z level=info msg="Executing migration" id="create team table" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.151274051Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.152167132Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.153020338Z level=info msg="Executing migration" id="create team member table" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.154038177Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.154836684Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.155722132Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.156492808Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.157356151Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T04:11:58.343 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.158243603Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.159148429Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.161852302Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.164846519Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.165409345Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.167649263Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.170580708Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.171232475Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.171699876Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.172170633Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.172582736Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.172895767Z level=info msg="Executing migration" id="create tag table" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.173310532Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.173789763Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T04:11:58.344 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.174205354Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.17463786Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.175110551Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.177041368Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.177459753Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.177927353Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.178250777Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.178598736Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.179032021Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.179503012Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.179820136Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.180537999Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.181308497Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.1821063Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T04:11:58.345 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.18291374Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T04:11:58.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.183359592Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T04:11:58.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.184133113Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T04:11:58.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.184537245Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T04:11:58.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.184998825Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T04:11:58.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.185425299Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T04:11:58.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.185883893Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T04:11:58.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.186347422Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T04:11:58.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.186797436Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T04:11:58.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.187566697Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T04:11:58.346 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.187992672Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T04:11:58.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.188478114Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T04:11:58.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.188940492Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T04:11:58.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.191783756Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T04:11:58.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.192296127Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T04:11:58.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.192584545Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T04:11:58.347 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.193065962Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T04:11:58.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.195771491Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T04:11:58.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.19865015Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T04:11:58.348 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.198960261Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T04:11:58.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.201127914Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T04:11:58.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.201659192Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T04:11:58.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.204135924Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T04:11:58.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.206997292Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T04:11:58.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.209629742Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T04:11:58.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.210079326Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T04:11:58.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.212906092Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T04:11:58.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.213372688Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T04:11:58.349 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.215744627Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.218345235Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.218660704Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.221270505Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.224074229Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.226772885Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.227456402Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.229970341Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.232568528Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.234966496Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.238350606Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.241691447Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.24446572Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.246993014Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T04:11:58.350 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.247810802Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T04:11:58.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.248252863Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T04:11:58.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.25086621Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T04:11:58.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.253644811Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T04:11:58.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.256254271Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T04:11:58.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.257156804Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T04:11:58.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.258228403Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T04:11:58.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.259082596Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T04:11:58.351 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.259933633Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T04:11:58.353 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.260786881Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T04:11:58.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.261743527Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T04:11:58.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.262659541Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T04:11:58.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.263536291Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T04:11:58.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.264352347Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T04:11:58.354 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.265220162Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.266140692Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.266933044Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.267693215Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.268720018Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.269562566Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.270280132Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.271185852Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.272077542Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.272944912Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.273806209Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.274689378Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T04:11:58.355 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.275703316Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T04:11:58.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.276609829Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T04:11:58.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.277611724Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T04:11:58.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.278533917Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T04:11:58.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.279444165Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T04:11:58.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.280276713Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T04:11:58.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.281177863Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T04:11:58.356 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.282056589Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T04:11:58.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.282959513Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T04:11:58.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.285804479Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T04:11:58.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.286213181Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T04:11:58.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.289004063Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T04:11:58.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.291932906Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T04:11:58.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.294615085Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T04:11:58.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.295107178Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T04:11:58.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.298023891Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T04:11:58.357 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.300855484Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.301013815Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.303437187Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.303830264Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.306746457Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.306865785Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.3069963Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.307396275Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.309603459Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.31250019Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.313065109Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.315938794Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.318599592Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T04:11:58.358 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.321238163Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T04:11:58.359 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.321616511Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T04:11:58.359 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.324550929Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T04:11:58.359 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.327027353Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T04:11:58.359 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.331368978Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T04:11:58.359 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.332288666Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T04:11:58.359 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.33504107Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T04:11:58.359 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.339236694Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T04:11:58.359 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.34352Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T04:11:58.359 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.34394898Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T04:11:58.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-26T04:11:58.710 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.346704743Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.349471674Z level=info msg="Executing migration" id="create permission table" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.352077319Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.35250454Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.355352058Z level=info msg="Executing migration" id="create role table" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.358169541Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.360676908Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.361575922Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.362428986Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.363278328Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.364264164Z level=info msg="Executing migration" id="create team role table" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.36467813Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.365555346Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T04:11:58.711 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.36640323Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T04:11:58.712 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.36734055Z level=info msg="Executing migration" id="create user role table" 2024-03-26T04:11:58.712 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.367829772Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T04:11:58.712 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.368673087Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T04:11:58.712 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.369452157Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T04:11:58.712 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.37037148Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T04:11:58.712 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.370932573Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T04:11:58.712 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.371890184Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T04:11:58.712 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.372545149Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T04:11:58.712 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.373540589Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T04:11:58.712 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.374034685Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.374862679Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.375735421Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.376743906Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.377286515Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.378182337Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.379231132Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.380008713Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.380479029Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.381433967Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.382282891Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.383009483Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.383409585Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.384293121Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T04:11:58.713 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.385145153Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T04:11:58.714 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.386026973Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T04:11:58.714 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.38648964Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T04:11:58.714 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.387448001Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T04:11:58.714 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.388292417Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.389261256Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.389897174Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.392668969Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.395414271Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.398337861Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.398890586Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.398981665Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.401821145Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.402002141Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.404469184Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.407246628Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.409951003Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.412690823Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:11:58.715 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.415604771Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.418530373Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.419004285Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.421753869Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.42225159Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.424798381Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.427504543Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.42799665Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.431055432Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.433906385Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.436623158Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.436967192Z level=info msg="Executing migration" id="add share column" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.439632851Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.442308534Z level=info msg="Executing migration" id="create file table" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.444746966Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T04:11:58.716 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.445219767Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.447852911Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.450625302Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.453061841Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.453406953Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.454244774Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.45514023Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.45559197Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.456623618Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.456965818Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.457850574Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.458768145Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.459734106Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.460280554Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T04:11:58.717 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.461344251Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.462371213Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.463211069Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.473055359Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.473898831Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.47472875Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.478306583Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.479144149Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.479995783Z level=info msg="Executing migration" id="create folder table" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.48089192Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.48176765Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.482256406Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.483045531Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T04:11:58.718 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=migrator t=2024-03-26T04:11:58.48353748Z level=info msg="migrations completed" performed=485 skipped=0 duration=739.689854ms 2024-03-26T04:11:58.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=sqlstore t=2024-03-26T04:11:58.483916817Z level=info msg="Created default organization" 2024-03-26T04:11:58.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=plugin.loader t=2024-03-26T04:11:58.540031053Z level=info msg="Plugin registered" pluginID=input 2024-03-26T04:11:58.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=plugin.loader t=2024-03-26T04:11:58.567194306Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-26T04:11:58.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=plugin.loader t=2024-03-26T04:11:58.567214086Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-26T04:11:58.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=secrets t=2024-03-26T04:11:58.56726737Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T04:11:58.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=query_data t=2024-03-26T04:11:58.569310664Z level=info msg="Query Service initialization" 2024-03-26T04:11:58.719 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=live.push_http t=2024-03-26T04:11:58.571699459Z level=info msg="Live Push Gateway initialization" 2024-03-26T04:11:58.792 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:58 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.792 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:58 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.792 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:58 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.792 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:58 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.792 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:58 smithi136 ceph-mon[25848]: Deploying daemon node-exporter.a on smithi136 2024-03-26T04:11:58.792 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:58 smithi136 ceph-mon[25848]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:58.793 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:58 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.806 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:58.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-mon[30657]: Deploying daemon node-exporter.a on smithi136 2024-03-26T04:11:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-mon[30657]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:58.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:58 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:58 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:58 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:58 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:58 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:58.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:58 smithi204 ceph-mon[30652]: Deploying daemon node-exporter.a on smithi136 2024-03-26T04:11:58.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:58 smithi204 ceph-mon[30652]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:58.998 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:58 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:11:59.397 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:11:59 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ::ffff:172.21.15.149 - - [26/Mar/2024:04:11:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:59.488 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=infra.usagestats.collector t=2024-03-26T04:11:59.19510125Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T04:11:59.488 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=provisioning.datasources t=2024-03-26T04:11:59.196105221Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T04:11:59.488 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=provisioning.datasources t=2024-03-26T04:11:59.207088649Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T04:11:59.488 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=provisioning.alerting t=2024-03-26T04:11:59.217807813Z level=info msg="starting to provision alerting" 2024-03-26T04:11:59.488 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=provisioning.alerting t=2024-03-26T04:11:59.217826231Z level=info msg="finished to provision alerting" 2024-03-26T04:11:59.488 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=ngalert.state.manager t=2024-03-26T04:11:59.218009175Z level=info msg="Warming state cache for startup" 2024-03-26T04:11:59.488 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=grafanaStorageLogger t=2024-03-26T04:11:59.218168641Z level=info msg="storage starting" 2024-03-26T04:11:59.488 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=http.server t=2024-03-26T04:11:59.219694986Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T04:11:59.488 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=ngalert.state.manager t=2024-03-26T04:11:59.220509031Z level=info msg="State cache has been initialized" states=0 duration=2.500826ms 2024-03-26T04:11:59.488 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=ticker t=2024-03-26T04:11:59.220571226Z level=info msg=starting first_tick=2024-03-26T04:12:00Z 2024-03-26T04:11:59.489 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=ngalert.multiorg.alertmanager t=2024-03-26T04:11:59.22058285Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T04:11:59.489 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=sqlstore.transactions t=2024-03-26T04:11:59.232343248Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T04:11:59.489 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=sqlstore.transactions t=2024-03-26T04:11:59.235923516Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T04:11:59.489 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a[42980]: logger=sqlstore.transactions t=2024-03-26T04:11:59.246399618Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-26T04:12:00.078 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:11:59 smithi149 ceph-mon[30657]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:00.143 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:11:59 smithi136 ceph-mon[25848]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:00.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:11:59 smithi204 ceph-mon[30652]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:00.883 INFO:teuthology.orchestra.run.smithi136.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-26T04:12:00.883 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/nvme0n1 ssd INTEL_SSDPEDMX020T4_CVPD514100122P0SGN 1863G No 2m ago Has a FileSystem, LVM detected 2024-03-26T04:12:00.883 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/nvme1n1 ssd Linux_c55a52c167e11ef647bb 89.4G Yes 2m ago 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/nvme2n1 ssd Linux_26648050adf6e08f910b 89.4G Yes 2m ago 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/nvme3n1 ssd Linux_912fe40b46b1581d44cc 89.4G Yes 2m ago 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/nvme4n1 ssd Linux_90199d68f5639f4a3d36 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi149 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CK400BGN 372G No 108s ago Has a FileSystem, LVM detected 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi149 /dev/nvme1n1 ssd Linux_9c2891eac819482b3938 89.4G Yes 108s ago 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi149 /dev/nvme2n1 ssd Linux_5cdf5ef71f9776f85b8a 89.4G Yes 108s ago 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi149 /dev/nvme3n1 ssd Linux_9dd9cdac63af18ec9d85 89.4G Yes 108s ago 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi149 /dev/nvme4n1 ssd Linux_17aeb1fa853bb1826a22 89.4G No 108s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi204 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GD400BGN 372G No 85s ago Has a FileSystem, LVM detected 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi204 /dev/nvme1n1 ssd Linux_34c6d14bf58602f019ac 89.4G Yes 85s ago 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi204 /dev/nvme2n1 ssd Linux_129627ed0070849689e9 89.4G Yes 85s ago 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi204 /dev/nvme3n1 ssd Linux_e66e068177209b774f9d 89.4G Yes 85s ago 2024-03-26T04:12:00.884 INFO:teuthology.orchestra.run.smithi136.stdout:smithi204 /dev/nvme4n1 ssd Linux_ebaae01b9eebefd29b2c 89.4G No 85s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T04:12:01.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:00 smithi136 bash[59367]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T04:12:02.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:01 smithi136 ceph-mon[25848]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:02.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:01 smithi136 ceph-mon[25848]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:02.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-26T04:12:02.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:01 smithi149 ceph-mon[30657]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:02.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:01 smithi149 ceph-mon[30657]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:02.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:01 smithi204 ceph-mon[30652]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:02.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:01 smithi204 ceph-mon[30652]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:02.392 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:02.698 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:02 smithi136 bash[59367]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T04:12:03.086 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:02 smithi136 bash[59367]: Writing manifest to image destination 2024-03-26T04:12:03.408 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 podman[59367]: 2024-03-26 04:12:03.17590088 +0000 UTC m=+2.932455303 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T04:12:03.408 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 podman[59367]: 2024-03-26 04:12:03.408341109 +0000 UTC m=+3.164895575 container create 2047947d50cb6679efb0bdd4b0b70dd7b39e5220a28b47896e34e7fa6cc86360 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:03.658 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 podman[59367]: 2024-03-26 04:12:03.652657813 +0000 UTC m=+3.409212237 container init 2047947d50cb6679efb0bdd4b0b70dd7b39e5220a28b47896e34e7fa6cc86360 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:03.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-mon[25848]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:03.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:03.921 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 podman[59367]: 2024-03-26 04:12:03.658889802 +0000 UTC m=+3.415444226 container start 2047947d50cb6679efb0bdd4b0b70dd7b39e5220a28b47896e34e7fa6cc86360 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.663Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.663Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T04:12:03.922 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T04:12:03.923 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T04:12:03.924 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T04:12:03.925 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.664Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T04:12:03.926 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.665Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T04:12:03.926 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a[59596]: ts=2024-03-26T04:12:03.665Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T04:12:03.926 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 bash[59367]: 2047947d50cb6679efb0bdd4b0b70dd7b39e5220a28b47896e34e7fa6cc86360 2024-03-26T04:12:03.926 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:03 smithi136 systemd[1]: Started Ceph node-exporter.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:12:04.237 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:03 smithi149 ceph-mon[30657]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:04.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:03 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:04.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:03 smithi204 ceph-mon[30652]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:04.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:03 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:04.937 INFO:teuthology.orchestra.run.smithi136.stdout:service_type: alertmanager 2024-03-26T04:12:04.937 INFO:teuthology.orchestra.run.smithi136.stdout:service_name: alertmanager 2024-03-26T04:12:04.937 INFO:teuthology.orchestra.run.smithi136.stdout:placement: 2024-03-26T04:12:04.937 INFO:teuthology.orchestra.run.smithi136.stdout: count: 1 2024-03-26T04:12:04.937 INFO:teuthology.orchestra.run.smithi136.stdout: hosts: 2024-03-26T04:12:04.937 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi136=a 2024-03-26T04:12:04.937 INFO:teuthology.orchestra.run.smithi136.stdout:status: 2024-03-26T04:12:04.937 INFO:teuthology.orchestra.run.smithi136.stdout: created: '2024-03-26T04:10:54.459991Z' 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout: ports: 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout: - 9093 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout: - 9094 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout: running: 0 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout: size: 1 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout:events: 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout:- 2024-03-26T04:11:44.131459Z service:alertmanager [INFO] "service was created" 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout:service_type: grafana 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout:service_name: grafana 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout:placement: 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout: count: 1 2024-03-26T04:12:04.938 INFO:teuthology.orchestra.run.smithi136.stdout: hosts: 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi149=a 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout:spec: 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout: anonymous_access: true 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout: protocol: https 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout:status: 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout: created: '2024-03-26T04:10:57.076214Z' 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout: ports: 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout: - 3000 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout: running: 0 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout: size: 1 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout:events: 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout:- 2024-03-26T04:11:57.753291Z service:grafana [INFO] "service was created" 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout:service_type: mgr 2024-03-26T04:12:04.939 INFO:teuthology.orchestra.run.smithi136.stdout:service_name: mgr 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout:placement: 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout: count: 3 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout: hosts: 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi136=y 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi149=x 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi204=z 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout:status: 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout: created: '2024-03-26T04:09:16.379905Z' 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout: last_refresh: '2024-03-26T04:11:19.918055Z' 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout: running: 3 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout: size: 3 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout:service_type: mon 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout:service_name: mon 2024-03-26T04:12:04.940 INFO:teuthology.orchestra.run.smithi136.stdout:placement: 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout: count: 3 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout: hosts: 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi136:172.21.15.136=a 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi149:172.21.15.149=b 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi204:172.21.15.204=c 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout:status: 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout: created: '2024-03-26T04:08:47.794904Z' 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout: last_refresh: '2024-03-26T04:11:19.917937Z' 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout: running: 3 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout: size: 3 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout:service_type: node-exporter 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout:service_name: node-exporter 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout:placement: 2024-03-26T04:12:04.941 INFO:teuthology.orchestra.run.smithi136.stdout: count: 3 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout: hosts: 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi136=a 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi149=b 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi204=c 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout:status: 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout: created: '2024-03-26T04:10:51.777931Z' 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout: ports: 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout: - 9100 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout: running: 0 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout: size: 3 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout:events: 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout:- 2024-03-26T04:12:03.835177Z service:node-exporter [INFO] "service was created" 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2024-03-26T04:12:04.942 INFO:teuthology.orchestra.run.smithi136.stdout:service_type: osd 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout:service_name: osd 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout:unmanaged: true 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout:spec: 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout: filter_logic: AND 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout: objectstore: bluestore 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout:status: 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout: container_image_id: 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout: last_refresh: '2024-03-26T04:11:19.918132Z' 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout: running: 3 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout: size: 3 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout:service_type: prometheus 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout:service_name: prometheus 2024-03-26T04:12:04.943 INFO:teuthology.orchestra.run.smithi136.stdout:placement: 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout: count: 1 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout: hosts: 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi149=a 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout:status: 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout: created: '2024-03-26T04:10:49.141958Z' 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout: last_refresh: '2024-03-26T04:11:20.675630Z' 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout: ports: 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout: - 9095 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout: running: 1 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout: size: 1 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout:service_type: rgw 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout:service_id: foo.a 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout:service_name: rgw.foo.a 2024-03-26T04:12:04.944 INFO:teuthology.orchestra.run.smithi136.stdout:placement: 2024-03-26T04:12:04.945 INFO:teuthology.orchestra.run.smithi136.stdout: count: 1 2024-03-26T04:12:04.945 INFO:teuthology.orchestra.run.smithi136.stdout: hosts: 2024-03-26T04:12:04.945 INFO:teuthology.orchestra.run.smithi136.stdout: - smithi136=foo.a 2024-03-26T04:12:04.945 INFO:teuthology.orchestra.run.smithi136.stdout:status: 2024-03-26T04:12:04.945 INFO:teuthology.orchestra.run.smithi136.stdout: created: '2024-03-26T04:10:50.183816Z' 2024-03-26T04:12:04.945 INFO:teuthology.orchestra.run.smithi136.stdout: last_refresh: '2024-03-26T04:11:27.839863Z' 2024-03-26T04:12:04.945 INFO:teuthology.orchestra.run.smithi136.stdout: ports: 2024-03-26T04:12:04.945 INFO:teuthology.orchestra.run.smithi136.stdout: - 80 2024-03-26T04:12:04.945 INFO:teuthology.orchestra.run.smithi136.stdout: running: 1 2024-03-26T04:12:04.945 INFO:teuthology.orchestra.run.smithi136.stdout: size: 1 2024-03-26T04:12:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:04 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:12:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:04 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:12:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:04 smithi136 ceph-mon[25848]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:12:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:04 smithi136 ceph-mon[25848]: Deploying daemon node-exporter.b on smithi149 2024-03-26T04:12:05.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:04 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:12:05.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:04 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:12:05.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:04 smithi149 ceph-mon[30657]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:12:05.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:04 smithi149 ceph-mon[30657]: Deploying daemon node-exporter.b on smithi149 2024-03-26T04:12:05.238 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:04 smithi149 bash[43254]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T04:12:05.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:04 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:12:05.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:04 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:12:05.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:04 smithi204 ceph-mon[30652]: from='mgr.14445 172.21.15.136:0/829872504' entity='mgr.y' 2024-03-26T04:12:05.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:04 smithi204 ceph-mon[30652]: Deploying daemon node-exporter.b on smithi149 2024-03-26T04:12:05.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-26T04:12:05.738 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:05 smithi149 bash[43254]: Getting image source signatures 2024-03-26T04:12:05.738 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:05 smithi149 bash[43254]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T04:12:05.738 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:05 smithi149 bash[43254]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T04:12:05.738 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:05 smithi149 bash[43254]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T04:12:05.801 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:06.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:05 smithi136 ceph-mon[25848]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:06.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:05 smithi136 ceph-mon[25848]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T04:12:06.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:05 smithi149 ceph-mon[30657]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:06.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:05 smithi149 ceph-mon[30657]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T04:12:06.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:05 smithi204 ceph-mon[30652]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:06.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:05 smithi204 ceph-mon[30652]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T04:12:07.832 INFO:teuthology.orchestra.run.smithi136.stdout:osd 3 47s ago - 2024-03-26T04:12:08.143 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:07 smithi136 ceph-mon[25848]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:08.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:07 smithi149 ceph-mon[30657]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:08.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:07 smithi204 ceph-mon[30652]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:12:08.572 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T04:12:08.582 INFO:tasks.cephadm:Teardown begin 2024-03-26T04:12:08.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:12:08.613 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:12:08.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:12:08.685 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T04:12:08.685 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:12:08.701 DEBUG:teuthology.orchestra.run.smithi149:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:12:08.717 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:12:08.743 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T04:12:08.743 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T04:12:08.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.a 2024-03-26T04:12:08.844 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:08 smithi136 systemd[1]: Stopping Ceph mon.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:08.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:08 smithi149 ceph-mon[30657]: from='client.14577 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:08.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:08 smithi204 ceph-mon[30652]: from='client.14577 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:09.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:08 smithi136 ceph-mon[25848]: from='client.14577 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:09.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:09 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a[25825]: 2024-03-26T04:12:09.048+0000 7f1149997700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:12:09.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:09 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a[25825]: 2024-03-26T04:12:09.048+0000 7f1149997700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T04:12:09.119 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:09 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:09] ENGINE Bus STOPPING 2024-03-26T04:12:09.120 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:09 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:12:09.120 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:09 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:09] ENGINE Bus STOPPED 2024-03-26T04:12:09.120 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:09 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:09] ENGINE Bus STARTING 2024-03-26T04:12:09.120 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:09 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: ::ffff:172.21.15.149 - - [26/Mar/2024:04:12:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:12:09.398 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:09 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:09] ENGINE Serving on http://:::9283 2024-03-26T04:12:09.398 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:09 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:09] ENGINE Bus STARTED 2024-03-26T04:12:09.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:09 smithi136 podman[60017]: 2024-03-26 04:12:09.223450089 +0000 UTC m=+0.292449244 container died b6196b95c1e4cf89b7adc52f127efcc5f708a5349fb7801cbb6c42b8f4f657df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-26T04:12:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:09 smithi136 podman[60017]: 2024-03-26 04:12:09.539108097 +0000 UTC m=+0.608107248 container cleanup b6196b95c1e4cf89b7adc52f127efcc5f708a5349fb7801cbb6c42b8f4f657df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0) 2024-03-26T04:12:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:09 smithi136 bash[60017]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:12:10.035 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 04:12:09 smithi136 podman[60047]: 2024-03-26 04:12:09.782267523 +0000 UTC m=+0.557383776 container remove b6196b95c1e4cf89b7adc52f127efcc5f708a5349fb7801cbb6c42b8f4f657df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T04:12:10.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T04:12:10.090 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:10.090 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T04:12:10.090 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-26T04:12:10.090 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.b 2024-03-26T04:12:10.488 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:10 smithi149 systemd[1]: Stopping Ceph mon.b for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:11.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:11 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:11] ENGINE Bus STOPPING 2024-03-26T04:12:11.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:11 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:11] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:12:11.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:11 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:11] ENGINE Bus STOPPED 2024-03-26T04:12:11.949 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:11 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:11] ENGINE Bus STARTING 2024-03-26T04:12:11.949 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:11 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:11] ENGINE Serving on http://:::9283 2024-03-26T04:12:11.949 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:11 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: [26/Mar/2024:04:12:11] ENGINE Bus STARTED 2024-03-26T04:12:11.987 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:11 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-b[30634]: 2024-03-26T04:12:11.614+0000 7fabe46c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:12:11.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:11 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-b[30634]: 2024-03-26T04:12:11.614+0000 7fabe46c9700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T04:12:11.988 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:11 smithi149 podman[43348]: 2024-03-26 04:12:11.666313121 +0000 UTC m=+1.387989057 container died a2d968c1e135db7c2e367062433b6d81fffeb5e3201fc1fc75b135b37f2739ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:12:13.238 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 bash[43254]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T04:12:13.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:13 smithi149 podman[43348]: 2024-03-26 04:12:13.112437085 +0000 UTC m=+2.834113021 container cleanup a2d968c1e135db7c2e367062433b6d81fffeb5e3201fc1fc75b135b37f2739ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T04:12:13.238 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:13 smithi149 bash[43348]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-b 2024-03-26T04:12:13.636 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 bash[43254]: Writing manifest to image destination 2024-03-26T04:12:13.637 INFO:journalctl@ceph.mon.b.smithi149.stdout:Mar 26 04:12:13 smithi149 podman[43367]: 2024-03-26 04:12:13.36413078 +0000 UTC m=+1.693286082 container remove a2d968c1e135db7c2e367062433b6d81fffeb5e3201fc1fc75b135b37f2739ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-26T04:12:13.887 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 podman[43254]: 2024-03-26 04:12:13.568702458 +0000 UTC m=+8.680376497 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T04:12:13.887 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 podman[43254]: 2024-03-26 04:12:13.684296766 +0000 UTC m=+8.795970846 container create 773a814b2f023a0b52deec49e43c5497e12dcd5745cde882829fd8e689af0389 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:12:13.887 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 podman[43254]: 2024-03-26 04:12:13.878305333 +0000 UTC m=+8.989979371 container init 773a814b2f023a0b52deec49e43c5497e12dcd5745cde882829fd8e689af0389 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:12:13.887 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 podman[43254]: 2024-03-26 04:12:13.883796491 +0000 UTC m=+8.995470531 container start 773a814b2f023a0b52deec49e43c5497e12dcd5745cde882829fd8e689af0389 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:12:14.010 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T04:12:14.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:14.091 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-26T04:12:14.092 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-26T04:12:14.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.c 2024-03-26T04:12:14.238 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.887Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.887Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.887Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.887Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T04:12:14.239 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T04:12:14.240 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T04:12:14.240 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T04:12:14.240 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T04:12:14.240 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T04:12:14.240 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T04:12:14.240 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T04:12:14.241 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T04:12:14.241 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T04:12:14.241 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T04:12:14.241 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T04:12:14.241 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T04:12:14.241 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T04:12:14.241 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T04:12:14.241 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T04:12:14.241 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T04:12:14.242 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T04:12:14.242 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T04:12:14.242 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T04:12:14.242 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T04:12:14.242 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T04:12:14.242 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T04:12:14.242 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T04:12:14.242 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.888Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T04:12:14.242 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T04:12:14.242 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T04:12:14.243 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T04:12:14.244 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T04:12:14.244 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T04:12:14.244 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T04:12:14.244 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b[43452]: ts=2024-03-26T04:12:13.889Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T04:12:14.244 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 bash[43254]: 773a814b2f023a0b52deec49e43c5497e12dcd5745cde882829fd8e689af0389 2024-03-26T04:12:14.244 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:12:13 smithi149 systemd[1]: Started Ceph node-exporter.b for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:12:14.407 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:14 smithi204 systemd[1]: Stopping Ceph mon.c for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:14.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:14 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-c[30629]: 2024-03-26T04:12:14.406+0000 7f6e00c28700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:12:14.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:14 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-c[30629]: 2024-03-26T04:12:14.406+0000 7f6e00c28700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T04:12:14.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:14 smithi204 podman[44110]: 2024-03-26 04:12:14.632481952 +0000 UTC m=+0.343712245 container died fce8ab718fac3875670bbf3bbfca54d3ba9583db33de3ac933fd8d9e38e9dc76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-c, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-26T04:12:15.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:15 smithi204 podman[44110]: 2024-03-26 04:12:15.129128195 +0000 UTC m=+0.840358531 container cleanup fce8ab718fac3875670bbf3bbfca54d3ba9583db33de3ac933fd8d9e38e9dc76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T04:12:15.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 26 04:12:15 smithi204 bash[44110]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mon-c 2024-03-26T04:12:15.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T04:12:15.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:15.518 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-26T04:12:15.518 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T04:12:15.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.y 2024-03-26T04:12:15.881 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:15 smithi136 systemd[1]: Stopping Ceph mgr.y for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:16.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:15 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:12:15.879+0000 7fea5271a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:12:16.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:15 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y[26102]: 2024-03-26T04:12:15.879+0000 7fea5271a700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-26T04:12:16.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:15 smithi136 podman[60180]: 2024-03-26 04:12:15.911903066 +0000 UTC m=+0.195975072 container died 06ee71789a25d6a906d05cee3000f11e71a06f1a2788c6bffb1a7bae4a4771e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:12:16.448 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:16 smithi136 podman[60180]: 2024-03-26 04:12:16.353493668 +0000 UTC m=+0.637565725 container cleanup 06ee71789a25d6a906d05cee3000f11e71a06f1a2788c6bffb1a7bae4a4771e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T04:12:16.448 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:16 smithi136 bash[60180]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y 2024-03-26T04:12:16.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T04:12:16.800 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:16 smithi136 podman[60200]: 2024-03-26 04:12:16.546583365 +0000 UTC m=+0.632007423 container remove 06ee71789a25d6a906d05cee3000f11e71a06f1a2788c6bffb1a7bae4a4771e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-26T04:12:16.800 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:16 smithi136 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.y.service: Deactivated successfully. 2024-03-26T04:12:16.800 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:16 smithi136 systemd[1]: Stopped Ceph mgr.y for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:12:16.801 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 04:12:16 smithi136 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.y.service: Consumed 51.765s CPU time. 2024-03-26T04:12:16.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:16.813 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T04:12:16.813 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T04:12:16.813 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.x 2024-03-26T04:12:17.137 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:12:16 smithi149 systemd[1]: Stopping Ceph mgr.x for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:17.431 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:12:17 smithi149 podman[43552]: 2024-03-26 04:12:17.137855685 +0000 UTC m=+0.133433538 container died f6579a30c564cbd589092cde15eb49e33734ab4743603b097c1bd1e711304be7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-26T04:12:17.737 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:12:17 smithi149 podman[43552]: 2024-03-26 04:12:17.513750498 +0000 UTC m=+0.509328351 container cleanup f6579a30c564cbd589092cde15eb49e33734ab4743603b097c1bd1e711304be7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:12:17.738 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Mar 26 04:12:17 smithi149 bash[43552]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-x 2024-03-26T04:12:17.932 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T04:12:17.969 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:17.969 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T04:12:17.969 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-26T04:12:17.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.z 2024-03-26T04:12:18.350 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:12:18 smithi204 systemd[1]: Stopping Ceph mgr.z for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:18.674 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:12:18 smithi204 podman[44241]: 2024-03-26 04:12:18.350580537 +0000 UTC m=+0.189206181 container died 857705d92f9366a52fa0593df7950e096ce4f70d269b9f03c0743e8d9bd3fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-26T04:12:18.997 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:12:18 smithi204 podman[44241]: 2024-03-26 04:12:18.674494775 +0000 UTC m=+0.513120462 container cleanup 857705d92f9366a52fa0593df7950e096ce4f70d269b9f03c0743e8d9bd3fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux ) 2024-03-26T04:12:18.997 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:12:18 smithi204 bash[44241]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z 2024-03-26T04:12:18.997 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:12:18 smithi204 podman[44260]: 2024-03-26 04:12:18.867101105 +0000 UTC m=+0.512071195 container remove 857705d92f9366a52fa0593df7950e096ce4f70d269b9f03c0743e8d9bd3fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T04:12:18.997 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 26 04:12:18 smithi204 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:12:19.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-26T04:12:19.128 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:19.128 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-26T04:12:19.128 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T04:12:19.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@osd.0 2024-03-26T04:12:19.418 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:19 smithi136 systemd[1]: Stopping Ceph osd.0 for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:19.698 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:19 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0[44214]: 2024-03-26T04:12:19.417+0000 7f3f50e47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:12:19.698 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:19 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0[44214]: 2024-03-26T04:12:19.417+0000 7f3f50e47700 -1 osd.0 31 *** Got signal Terminated *** 2024-03-26T04:12:19.698 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:19 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0[44214]: 2024-03-26T04:12:19.417+0000 7f3f50e47700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:12:25.067 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:24 smithi136 podman[60312]: 2024-03-26 04:12:24.767264572 +0000 UTC m=+5.447733326 container died cccd09bd51a8d81c7e60f1c7592165964117a880c7e1d7a6029a58970ef6671f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T04:12:25.448 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:25 smithi136 podman[60312]: 2024-03-26 04:12:25.249045031 +0000 UTC m=+5.929513791 container cleanup cccd09bd51a8d81c7e60f1c7592165964117a880c7e1d7a6029a58970ef6671f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-26T04:12:25.448 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:25 smithi136 bash[60312]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0 2024-03-26T04:12:25.775 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:25 smithi136 podman[60334]: 2024-03-26 04:12:25.475227461 +0000 UTC m=+0.700352595 container remove cccd09bd51a8d81c7e60f1c7592165964117a880c7e1d7a6029a58970ef6671f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-26T04:12:26.086 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:25 smithi136 podman[60429]: 2024-03-26 04:12:25.776026747 +0000 UTC m=+0.107057072 container create 4521adf0a99f9f1242a04fb058a47299849a34e3f0b74c9f8880dbd8ed9bf62c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:12:26.086 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:25 smithi136 podman[60429]: 2024-03-26 04:12:25.686514383 +0000 UTC m=+0.017544669 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:12:26.086 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:25 smithi136 podman[60429]: 2024-03-26 04:12:25.969841758 +0000 UTC m=+0.300872030 container init 4521adf0a99f9f1242a04fb058a47299849a34e3f0b74c9f8880dbd8ed9bf62c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T04:12:26.086 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:25 smithi136 podman[60429]: 2024-03-26 04:12:25.975335323 +0000 UTC m=+0.306365595 container start 4521adf0a99f9f1242a04fb058a47299849a34e3f0b74c9f8880dbd8ed9bf62c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T04:12:26.086 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:26 smithi136 podman[60429]: 2024-03-26 04:12:26.086440237 +0000 UTC m=+0.417470512 container attach 4521adf0a99f9f1242a04fb058a47299849a34e3f0b74c9f8880dbd8ed9bf62c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-26T04:12:26.448 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 04:12:26 smithi136 podman[60429]: 2024-03-26 04:12:26.268032152 +0000 UTC m=+0.599062425 container died 4521adf0a99f9f1242a04fb058a47299849a34e3f0b74c9f8880dbd8ed9bf62c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=) 2024-03-26T04:12:26.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T04:12:26.955 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:26.955 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T04:12:26.955 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T04:12:26.955 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@osd.1 2024-03-26T04:12:27.286 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:27 smithi149 systemd[1]: Stopping Ceph osd.1 for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:27.738 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:27 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1[35231]: 2024-03-26T04:12:27.285+0000 7efdf9187700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:12:27.738 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:27 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1[35231]: 2024-03-26T04:12:27.285+0000 7efdf9187700 -1 osd.1 31 *** Got signal Terminated *** 2024-03-26T04:12:27.738 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:27 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1[35231]: 2024-03-26T04:12:27.285+0000 7efdf9187700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:12:33.021 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:32 smithi149 podman[43695]: 2024-03-26 04:12:32.752006083 +0000 UTC m=+5.580718155 container died a52de1a6605b2cc14d722392b2c746899ed8a54815f328ac9b8dc98ccecfa8af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T04:12:33.021 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:33 smithi149 podman[43695]: 2024-03-26 04:12:33.021179034 +0000 UTC m=+5.849891103 container cleanup a52de1a6605b2cc14d722392b2c746899ed8a54815f328ac9b8dc98ccecfa8af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T04:12:33.393 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:33 smithi149 bash[43695]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1 2024-03-26T04:12:33.737 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:33 smithi149 podman[43718]: 2024-03-26 04:12:33.393967636 +0000 UTC m=+0.642565023 container remove a52de1a6605b2cc14d722392b2c746899ed8a54815f328ac9b8dc98ccecfa8af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T04:12:34.088 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:33 smithi149 podman[43815]: 2024-03-26 04:12:33.786323071 +0000 UTC m=+0.110847713 container create 1019bf9af417636a6474292b2bb5f7ea353466b252123f002cb4ff0fac581524 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T04:12:34.088 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:33 smithi149 podman[43815]: 2024-03-26 04:12:33.693692286 +0000 UTC m=+0.018216928 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:12:34.089 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:33 smithi149 podman[43815]: 2024-03-26 04:12:33.996962759 +0000 UTC m=+0.321487401 container init 1019bf9af417636a6474292b2bb5f7ea353466b252123f002cb4ff0fac581524 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T04:12:34.089 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:34 smithi149 podman[43815]: 2024-03-26 04:12:34.002703786 +0000 UTC m=+0.327228431 container start 1019bf9af417636a6474292b2bb5f7ea353466b252123f002cb4ff0fac581524 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T04:12:34.089 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:34 smithi149 podman[43815]: 2024-03-26 04:12:34.0884183 +0000 UTC m=+0.412942978 container attach 1019bf9af417636a6474292b2bb5f7ea353466b252123f002cb4ff0fac581524 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-26T04:12:34.488 INFO:journalctl@ceph.osd.1.smithi149.stdout:Mar 26 04:12:34 smithi149 podman[43815]: 2024-03-26 04:12:34.298453198 +0000 UTC m=+0.622977833 container died 1019bf9af417636a6474292b2bb5f7ea353466b252123f002cb4ff0fac581524 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-26T04:12:34.856 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T04:12:34.896 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:34.896 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T04:12:34.896 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T04:12:34.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@osd.2 2024-03-26T04:12:35.196 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:34 smithi204 systemd[1]: Stopping Ceph osd.2 for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:35.497 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:35 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2[35910]: 2024-03-26T04:12:35.195+0000 7fbfb4198700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:12:35.497 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:35 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2[35910]: 2024-03-26T04:12:35.195+0000 7fbfb4198700 -1 osd.2 31 *** Got signal Terminated *** 2024-03-26T04:12:35.497 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:35 smithi204 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2[35910]: 2024-03-26T04:12:35.195+0000 7fbfb4198700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:12:40.794 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:40 smithi204 podman[44376]: 2024-03-26 04:12:40.505696815 +0000 UTC m=+5.415209255 container died 18f0a4b504d494bbf578ee78601578cea477a2d159541fdb51e4e437198a887b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:12:41.118 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:40 smithi204 podman[44376]: 2024-03-26 04:12:40.892938579 +0000 UTC m=+5.802451030 container cleanup 18f0a4b504d494bbf578ee78601578cea477a2d159541fdb51e4e437198a887b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-03-26T04:12:41.118 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:40 smithi204 bash[44376]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2 2024-03-26T04:12:41.410 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:41 smithi204 podman[44399]: 2024-03-26 04:12:41.11862889 +0000 UTC m=+0.611751584 container remove 18f0a4b504d494bbf578ee78601578cea477a2d159541fdb51e4e437198a887b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:12:41.749 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:41 smithi204 podman[44492]: 2024-03-26 04:12:41.411091932 +0000 UTC m=+0.095052234 container create 2831ac5ecf673687a355d6d920e04a41e16ec355433fa0baa4a24a1ec05d0dc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-26T04:12:41.750 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:41 smithi204 podman[44492]: 2024-03-26 04:12:41.333944862 +0000 UTC m=+0.017905177 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:12:41.750 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:41 smithi204 podman[44492]: 2024-03-26 04:12:41.572047947 +0000 UTC m=+0.256008249 container init 2831ac5ecf673687a355d6d920e04a41e16ec355433fa0baa4a24a1ec05d0dc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T04:12:41.750 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:41 smithi204 podman[44492]: 2024-03-26 04:12:41.57784892 +0000 UTC m=+0.261809223 container start 2831ac5ecf673687a355d6d920e04a41e16ec355433fa0baa4a24a1ec05d0dc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T04:12:41.750 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:41 smithi204 podman[44492]: 2024-03-26 04:12:41.655019124 +0000 UTC m=+0.338979441 container attach 2831ac5ecf673687a355d6d920e04a41e16ec355433fa0baa4a24a1ec05d0dc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T04:12:42.165 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 26 04:12:41 smithi204 podman[44492]: 2024-03-26 04:12:41.863950163 +0000 UTC m=+0.547910469 container died 2831ac5ecf673687a355d6d920e04a41e16ec355433fa0baa4a24a1ec05d0dc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:12:42.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T04:12:42.491 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:42.491 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T04:12:42.491 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T04:12:42.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T04:12:42.793 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:42 smithi136 systemd[1]: Stopping Ceph rgw.foo.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:43.194 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:42 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-rgw-foo-a[50517]: 2024-03-26T04:12:42.792+0000 7f842fa45700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:12:43.194 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:42 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-rgw-foo-a[50517]: 2024-03-26T04:12:42.792+0000 7f843e2c2ac0 -1 shutting down 2024-03-26T04:12:53.107 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:52 smithi136 bash[60561]: time="2024-03-26T04:12:52Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T04:12:53.107 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:52 smithi136 podman[60561]: 2024-03-26 04:12:52.815092237 +0000 UTC m=+10.123958956 container died d4d28675bacd9d50ca1e81ffb940ea293bdf2eca1d2d7beb89f60bb283bde269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-rgw-foo-a, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-03-26T04:12:53.425 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:53 smithi136 podman[60561]: 2024-03-26 04:12:53.297321554 +0000 UTC m=+10.606188271 container cleanup d4d28675bacd9d50ca1e81ffb940ea293bdf2eca1d2d7beb89f60bb283bde269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-26T04:12:53.425 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:53 smithi136 bash[60561]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T04:12:53.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T04:12:53.690 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:53 smithi136 podman[60598]: 2024-03-26 04:12:53.531776574 +0000 UTC m=+0.707911067 container remove d4d28675bacd9d50ca1e81ffb940ea293bdf2eca1d2d7beb89f60bb283bde269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T04:12:53.690 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:53 smithi136 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T04:12:53.690 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:53 smithi136 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-26T04:12:53.690 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:53 smithi136 systemd[1]: Stopped Ceph rgw.foo.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:12:53.690 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 04:12:53 smithi136 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@rgw.foo.a.service: Consumed 1.427s CPU time. 2024-03-26T04:12:53.703 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:53.704 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T04:12:53.704 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T04:12:53.704 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T04:12:53.987 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 systemd[1]: Stopping Ceph prometheus.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:54.275 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.994Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.994Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.994Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.994Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.994Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.994Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.994Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.994Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.994Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.995Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.995Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:53 smithi149 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a[41104]: ts=2024-03-26T04:12:53.995Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T04:12:54.276 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:54 smithi149 podman[43946]: 2024-03-26 04:12:54.0054638 +0000 UTC m=+0.117073301 container died 669c70d350055e7f287863846c1cd58eb12a6c43a1cbea655e255a1e40b24a01 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:54.583 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:54 smithi149 podman[43946]: 2024-03-26 04:12:54.383394173 +0000 UTC m=+0.495003667 container cleanup 669c70d350055e7f287863846c1cd58eb12a6c43a1cbea655e255a1e40b24a01 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:54.584 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Mar 26 04:12:54 smithi149 bash[43946]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T04:12:54.731 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T04:12:54.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:12:54.769 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T04:12:54.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:12:54.955 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:12:55.948 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:12:55 smithi136 systemd[1]: Stopping Ceph alertmanager.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:56.369 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:12:56 smithi136 ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a[57872]: ts=2024-03-26T04:12:56.054Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:12:56.370 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:12:56 smithi136 podman[60917]: 2024-03-26 04:12:56.085584362 +0000 UTC m=+0.158472819 container died 21621393e316a6b6ef7cf27cc564c63ee86484d75e53c2ae9dcb70a62f774ae8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:56.698 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:12:56 smithi136 podman[60917]: 2024-03-26 04:12:56.543851805 +0000 UTC m=+0.616740307 container cleanup 21621393e316a6b6ef7cf27cc564c63ee86484d75e53c2ae9dcb70a62f774ae8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:56.698 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:12:56 smithi136 bash[60917]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T04:12:57.198 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:12:56 smithi136 podman[60936]: 2024-03-26 04:12:56.794730238 +0000 UTC m=+0.713002883 container remove 21621393e316a6b6ef7cf27cc564c63ee86484d75e53c2ae9dcb70a62f774ae8 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:57.198 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:12:56 smithi136 podman[60936]: 2024-03-26 04:12:56.887137362 +0000 UTC m=+0.805410001 volume remove 89d5923af62e0e737dfe32f978dd7096b28f4b8895c7dfd786a2cbd1e8371667 2024-03-26T04:12:57.198 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:12:57 smithi136 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T04:12:57.198 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 04:12:57 smithi136 systemd[1]: Stopped Ceph alertmanager.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:12:57.478 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:57 smithi136 systemd[1]: Stopping Ceph node-exporter.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:12:57.872 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:57 smithi136 podman[61054]: 2024-03-26 04:12:57.478218217 +0000 UTC m=+0.121532051 container died 2047947d50cb6679efb0bdd4b0b70dd7b39e5220a28b47896e34e7fa6cc86360 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:57.872 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:57 smithi136 podman[61054]: 2024-03-26 04:12:57.873049326 +0000 UTC m=+0.516363159 container cleanup 2047947d50cb6679efb0bdd4b0b70dd7b39e5220a28b47896e34e7fa6cc86360 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:58.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:57 smithi136 bash[61054]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T04:12:58.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:58 smithi136 podman[61070]: 2024-03-26 04:12:58.107500534 +0000 UTC m=+0.630852639 container remove 2047947d50cb6679efb0bdd4b0b70dd7b39e5220a28b47896e34e7fa6cc86360 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:12:58.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:58 smithi136 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:12:58.632 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:58 smithi136 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T04:12:58.632 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:58 smithi136 systemd[1]: Stopped Ceph node-exporter.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:12:58.632 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 04:12:58 smithi136 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 1.725s CPU time. 2024-03-26T04:12:58.879 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:12:59.067 INFO:teuthology.orchestra.run.smithi149.stdout:Deleting cluster with fsid: 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:00.445 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:13:00 smithi149 systemd[1]: Stopping Ceph grafana.a for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:13:00.446 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:13:00 smithi149 podman[44297]: 2024-03-26 04:13:00.258744077 +0000 UTC m=+0.093832463 container died e6ff275b36d8df7384c111b257500a0dd8dc2f4215378fd7c0b4ccb082dbbd63 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, io.openshift.tags=base rhel8, version=8.5, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, distribution-scope=public, io.buildah.version=1.32.0, architecture=x86_64, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-26T04:13:00.738 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:13:00 smithi149 podman[44297]: 2024-03-26 04:13:00.503295908 +0000 UTC m=+0.338384290 container cleanup e6ff275b36d8df7384c111b257500a0dd8dc2f4215378fd7c0b4ccb082dbbd63 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, description=Ceph Grafana Container, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, release=239.1651231664, version=8.5, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-26T04:13:00.738 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:13:00 smithi149 bash[44297]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a 2024-03-26T04:13:00.738 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:13:00 smithi149 podman[44316]: 2024-03-26 04:13:00.646231202 +0000 UTC m=+0.385513307 container remove e6ff275b36d8df7384c111b257500a0dd8dc2f4215378fd7c0b4ccb082dbbd63 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-grafana-a, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, version=8.5, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, maintainer=Paul Cuzner , vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, release=239.1651231664, description=Ceph Grafana Container, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=) 2024-03-26T04:13:00.738 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:13:00 smithi149 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:13:01.238 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:13:01 smithi149 systemd[1]: Stopping Ceph node-exporter.b for 276a1e6c-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:13:01.238 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:13:00 smithi149 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T04:13:01.238 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:13:00 smithi149 systemd[1]: Stopped Ceph grafana.a for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:13:01.238 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Mar 26 04:13:00 smithi149 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@grafana.a.service: Consumed 2.791s CPU time. 2024-03-26T04:13:01.616 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:13:01 smithi149 podman[44433]: 2024-03-26 04:13:01.337345064 +0000 UTC m=+0.131730993 container died 773a814b2f023a0b52deec49e43c5497e12dcd5745cde882829fd8e689af0389 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:13:01.941 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:13:01 smithi149 podman[44433]: 2024-03-26 04:13:01.715611779 +0000 UTC m=+0.509997705 container cleanup 773a814b2f023a0b52deec49e43c5497e12dcd5745cde882829fd8e689af0389 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:13:01.941 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:13:01 smithi149 bash[44433]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T04:13:02.226 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:13:01 smithi149 podman[44452]: 2024-03-26 04:13:01.941535876 +0000 UTC m=+0.609033844 container remove 773a814b2f023a0b52deec49e43c5497e12dcd5745cde882829fd8e689af0389 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-276a1e6c-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:13:02.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:13:01 smithi149 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:13:02.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:13:02 smithi149 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T04:13:02.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:13:02 smithi149 systemd[1]: Stopped Ceph node-exporter.b for 276a1e6c-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:13:02.227 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Mar 26 04:13:02 smithi149 systemd[1]: ceph-276a1e6c-eb26-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 1.753s CPU time. 2024-03-26T04:13:02.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:13:02.923 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:04.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:13:04.126 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:13:04.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:13:04.191 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T04:13:04.193 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985/remote/smithi136/crash 2024-03-26T04:13:04.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:13:04.270 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:13:04.270 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:13:04.272 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985/remote/smithi149/crash 2024-03-26T04:13:04.273 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:13:04.310 INFO:teuthology.orchestra.run.smithi149.stderr:tar: /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:13:04.310 INFO:teuthology.orchestra.run.smithi149.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:13:04.312 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985/remote/smithi204/crash 2024-03-26T04:13:04.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:13:04.359 INFO:teuthology.orchestra.run.smithi204.stderr:tar: /var/lib/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:13:04.359 INFO:teuthology.orchestra.run.smithi204.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:13:04.360 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T04:13:04.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:13:04.393 INFO:teuthology.orchestra.run.smithi136.stdout:2024-03-26T04:09:07.276203+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:13:04.393 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T04:13:04.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[SEC\]' /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:13:04.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[ERR\]' /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:13:04.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[WRN\]' /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:13:04.602 INFO:teuthology.orchestra.run.smithi136.stdout:2024-03-26T04:09:07.276203+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:13:04.602 INFO:tasks.cephadm:Compressing logs... 2024-03-26T04:13:04.603 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:13:04.644 DEBUG:teuthology.orchestra.run.smithi149:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:13:04.647 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:13:04.669 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:13:04.670 INFO:teuthology.orchestra.run.smithi136.stderr:: No such file or directory 2024-03-26T04:13:04.670 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T04:13:04.670 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:13:04.670 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T04:13:04.670 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mon.a.log: /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T04:13:04.671 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T04:13:04.671 INFO:teuthology.orchestra.run.smithi136.stderr: 57.6% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T04:13:04.671 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:13:04.671 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mgr.y.log: /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:13:04.672 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:13:04.672 INFO:teuthology.orchestra.run.smithi136.stderr: 90.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:13:04.673 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:13:04.673 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log: 82.7% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:13:04.673 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-volume.log: 85.0% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr:: No such file or directory 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr: 90.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr: 81.3% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr: 89.5% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- 84.7% /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mon.c.log -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:13:04.677 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-26T04:13:04.678 INFO:teuthology.orchestra.run.smithi136.stderr: 90.4% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:13:04.679 INFO:teuthology.orchestra.run.smithi149.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-03-26T04:13:04.679 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-03-26T04:13:04.679 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:13:04.679 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T04:13:04.679 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T04:13:04.680 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:13:04.680 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mon.c.log: 90.2% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-26T04:13:04.681 INFO:teuthology.orchestra.run.smithi204.stderr: 90.5% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:13:04.682 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:13:04.682 INFO:teuthology.orchestra.run.smithi149.stderr: 89.4% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:13:04.682 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:13:04.682 INFO:teuthology.orchestra.run.smithi149.stderr: 81.3% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:13:04.682 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T04:13:04.682 INFO:teuthology.orchestra.run.smithi149.stderr: 81.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:13:04.682 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-volume.log: 84.5% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:13:04.683 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mon.b.log: 90.5% 90.2% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-volume.log.gz -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T04:13:04.683 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-03-26T04:13:04.728 INFO:teuthology.orchestra.run.smithi149.stderr: 92.3% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T04:13:04.731 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T04:13:04.745 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T04:13:04.851 INFO:teuthology.orchestra.run.smithi136.stderr: 91.3% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T04:13:04.954 INFO:teuthology.orchestra.run.smithi136.stderr: 94.2% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T04:13:04.955 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-26T04:13:04.956 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m0.299s 2024-03-26T04:13:04.956 INFO:teuthology.orchestra.run.smithi136.stderr:user 0m0.538s 2024-03-26T04:13:04.956 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m0.041s 2024-03-26T04:13:04.971 INFO:teuthology.orchestra.run.smithi204.stderr: 94.2% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T04:13:04.972 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-26T04:13:04.972 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m0.310s 2024-03-26T04:13:04.973 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m0.357s 2024-03-26T04:13:04.973 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m0.032s 2024-03-26T04:13:04.992 INFO:teuthology.orchestra.run.smithi149.stderr: 94.2% -- replaced with /var/log/ceph/276a1e6c-eb26-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T04:13:04.994 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-03-26T04:13:04.994 INFO:teuthology.orchestra.run.smithi149.stderr:real 0m0.332s 2024-03-26T04:13:04.994 INFO:teuthology.orchestra.run.smithi149.stderr:user 0m0.367s 2024-03-26T04:13:04.994 INFO:teuthology.orchestra.run.smithi149.stderr:sys 0m0.048s 2024-03-26T04:13:04.994 INFO:tasks.cephadm:Archiving logs... 2024-03-26T04:13:04.994 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985/remote/smithi136/log 2024-03-26T04:13:04.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:13:05.120 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985/remote/smithi149/log 2024-03-26T04:13:05.121 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:13:05.214 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985/remote/smithi204/log 2024-03-26T04:13:05.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:13:05.302 INFO:tasks.cephadm:Removing cluster... 2024-03-26T04:13:05.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 --force 2024-03-26T04:13:05.483 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:05.596 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 --force 2024-03-26T04:13:05.782 INFO:teuthology.orchestra.run.smithi149.stdout:Deleting cluster with fsid: 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:05.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 276a1e6c-eb26-11ee-95d0-87774f69a715 --force 2024-03-26T04:13:06.084 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: 276a1e6c-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:06.199 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T04:13:06.199 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T04:13:06.215 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T04:13:06.230 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T04:13:06.244 INFO:tasks.cephadm:Teardown complete 2024-03-26T04:13:06.245 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-26T04:13:06.254 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi136:/dev/vg_nvme/lv_1... 2024-03-26T04:13:06.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo nvme disconnect -n lv_1 2024-03-26T04:13:06.393 INFO:teuthology.orchestra.run.smithi136.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T04:13:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:06.395 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi136:/dev/vg_nvme/lv_2... 2024-03-26T04:13:06.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo nvme disconnect -n lv_2 2024-03-26T04:13:06.546 INFO:teuthology.orchestra.run.smithi136.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T04:13:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:06.547 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi136:/dev/vg_nvme/lv_3... 2024-03-26T04:13:06.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo nvme disconnect -n lv_3 2024-03-26T04:13:06.704 INFO:teuthology.orchestra.run.smithi136.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T04:13:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:06.706 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi136:/dev/vg_nvme/lv_4... 2024-03-26T04:13:06.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo nvme disconnect -n lv_4 2024-03-26T04:13:06.833 INFO:teuthology.orchestra.run.smithi136.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T04:13:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:06.835 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T04:13:06.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/scratch_devs 2024-03-26T04:13:06.861 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi149:/dev/vg_nvme/lv_1... 2024-03-26T04:13:06.861 DEBUG:teuthology.orchestra.run.smithi149:> sudo nvme disconnect -n lv_1 2024-03-26T04:13:07.010 INFO:teuthology.orchestra.run.smithi149.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T04:13:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:07.012 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi149:/dev/vg_nvme/lv_2... 2024-03-26T04:13:07.012 DEBUG:teuthology.orchestra.run.smithi149:> sudo nvme disconnect -n lv_2 2024-03-26T04:13:07.150 INFO:teuthology.orchestra.run.smithi149.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T04:13:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:07.151 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi149:/dev/vg_nvme/lv_3... 2024-03-26T04:13:07.152 DEBUG:teuthology.orchestra.run.smithi149:> sudo nvme disconnect -n lv_3 2024-03-26T04:13:07.282 INFO:teuthology.orchestra.run.smithi149.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T04:13:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:07.283 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi149:/dev/vg_nvme/lv_4... 2024-03-26T04:13:07.284 DEBUG:teuthology.orchestra.run.smithi149:> sudo nvme disconnect -n lv_4 2024-03-26T04:13:07.430 INFO:teuthology.orchestra.run.smithi149.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T04:13:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:07.432 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-03-26T04:13:07.432 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/scratch_devs 2024-03-26T04:13:07.462 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi204:/dev/vg_nvme/lv_1... 2024-03-26T04:13:07.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo nvme disconnect -n lv_1 2024-03-26T04:13:07.601 INFO:teuthology.orchestra.run.smithi204.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T04:13:07.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:07.603 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi204:/dev/vg_nvme/lv_2... 2024-03-26T04:13:07.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo nvme disconnect -n lv_2 2024-03-26T04:13:07.739 INFO:teuthology.orchestra.run.smithi204.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T04:13:07.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:07.741 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi204:/dev/vg_nvme/lv_3... 2024-03-26T04:13:07.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo nvme disconnect -n lv_3 2024-03-26T04:13:07.876 INFO:teuthology.orchestra.run.smithi204.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T04:13:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:07.878 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi204:/dev/vg_nvme/lv_4... 2024-03-26T04:13:07.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo nvme disconnect -n lv_4 2024-03-26T04:13:08.015 INFO:teuthology.orchestra.run.smithi204.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T04:13:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:08.017 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-26T04:13:08.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/scratch_devs 2024-03-26T04:13:08.049 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T04:13:08.058 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T04:13:08.059 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:13:08.061 DEBUG:teuthology.orchestra.run.smithi149:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:13:08.064 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:13:08.073 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:13:08.078 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:13:08.078 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-26T04:13:08.078 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 +286us[ +286us] +/- 73ms 2024-03-26T04:13:08.078 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -527us[ -527us] +/- 55ms 2024-03-26T04:13:08.078 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +483us[ +483us] +/- 74ms 2024-03-26T04:13:08.078 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:13:08.079 INFO:teuthology.orchestra.run.smithi149.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:13:08.084 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:13:08.084 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-03-26T04:13:08.084 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 41 +272us[ +272us] +/- 73ms 2024-03-26T04:13:08.084 INFO:teuthology.orchestra.run.smithi149.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -517us[ -502us] +/- 55ms 2024-03-26T04:13:08.084 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +476us[ +476us] +/- 74ms 2024-03-26T04:13:08.084 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:13:08.106 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:13:08.111 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:13:08.111 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-26T04:13:08.111 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 +267us[ +267us] +/- 73ms 2024-03-26T04:13:08.111 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -526us[ -516us] +/- 55ms 2024-03-26T04:13:08.111 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +487us[ +497us] +/- 74ms 2024-03-26T04:13:08.111 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:13:08.112 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T04:13:08.121 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T04:13:08.122 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T04:13:08.174 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:13:08.246 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:13:08.321 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:13:08.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:13:08.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:08.425 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:13:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:08.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:13:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:08.488 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T04:13:08.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T04:13:08.549 INFO:teuthology.task.internal:Duration was 709.758748 seconds 2024-03-26T04:13:08.549 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T04:13:08.616 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T04:13:08.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:13:08.619 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:13:08.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:13:08.660 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:13:08.664 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:13:08.669 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:13:09.047 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T04:13:09.047 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-03-26T04:13:09.047 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:13:09.066 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2024-03-26T04:13:09.067 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:13:09.092 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-03-26T04:13:09.092 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:13:09.115 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T04:13:09.116 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:13:09.118 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:13:09.135 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:13:09.185 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T04:13:09.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:13:09.187 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:13:09.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:13:09.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T04:13:09.387 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T04:13:09.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:13:09.414 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:13:09.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:13:09.468 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T04:13:09.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:13:09.481 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:13:09.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:13:09.503 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-03-26T04:13:09.507 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2024-03-26T04:13:09.534 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-03-26T04:13:09.567 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:13:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:09.581 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:13:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:09.607 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:13:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:13:09.622 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T04:13:09.632 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T04:13:09.633 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985/remote/smithi136 2024-03-26T04:13:09.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:13:09.671 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985/remote/smithi149 2024-03-26T04:13:09.671 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:13:09.707 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985/remote/smithi204 2024-03-26T04:13:09.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:13:09.743 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T04:13:09.743 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:13:09.745 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:13:09.747 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:13:09.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T04:13:09.803 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T04:13:09.803 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T04:13:09.841 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T04:13:09.841 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:13:09.844 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:13:09.846 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:13:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:13 /home/ubuntu/cephtest 2024-03-26T04:13:09.858 INFO:teuthology.orchestra.run.smithi149.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:13 /home/ubuntu/cephtest 2024-03-26T04:13:09.860 INFO:teuthology.orchestra.run.smithi204.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:13 /home/ubuntu/cephtest 2024-03-26T04:13:09.861 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T04:13:09.916 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T04:13:10.051 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T04:13:10.078 DEBUG:teuthology.nuke:shortname: smithi136 2024-03-26T04:13:10.078 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:13:10.105 DEBUG:teuthology.nuke:shortname: smithi149 2024-03-26T04:13:10.105 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:13:10.110 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:51:33.179160', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPHjTUqpJSdOkZX47TJiip8kRwGlE9H0aSdHyJDR4DlNZh3CHcg0TlTvv/X7xY/WwlAKTslNdktBxRX5pVtKSCg='} 2024-03-26T04:13:10.128 DEBUG:teuthology.nuke:shortname: smithi204 2024-03-26T04:13:10.128 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:13:10.131 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:51:33.178306', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2FUuQaQr44IF38rP8jG19B6CXq+9A3huVRWHjs/gnGGNx6xov3bC7mcPK+S/ZqWV9pMfRBRAAn83lYFe1t0ng='} 2024-03-26T04:13:10.146 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621985', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:51:33.177273', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRA5xmGNfBUGCo9IzqoIoh55PFwHg9rn7/uVloMm9FqMjywdxIhxjS9jV//qAdSyhfTYm9UDMGjNmavbEeaCiA='} 2024-03-26T04:13:10.159 INFO:teuthology.orchestra.console.smithi136:Power off 2024-03-26T04:13:10.159 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:13:10.184 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2024-03-26T04:13:10.184 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:13:10.199 INFO:teuthology.orchestra.console.smithi149:Power off 2024-03-26T04:13:10.200 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:13:10.214 INFO:teuthology.orchestra.console.smithi204:Power off 2024-03-26T04:13:10.214 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:13:10.222 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-26T04:13:10.223 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2024-03-26T04:13:10.223 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:13:10.238 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2024-03-26T04:13:10.238 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:13:10.246 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-03-26T04:13:10.262 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-26T04:13:14.223 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:13:14.247 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:13:14.264 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:13:14.333 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-26T04:13:14.360 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-03-26T04:13:14.372 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-26T04:13:18.335 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:13:18.361 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:13:18.373 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:13:18.445 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-03-26T04:13:18.470 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2024-03-26T04:13:18.482 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-26T04:13:18.546 INFO:teuthology.orchestra.console.smithi136:Power off completed 2024-03-26T04:13:18.571 INFO:teuthology.orchestra.console.smithi149:Power off completed 2024-03-26T04:13:22.484 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:13:22.595 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2024-03-26T04:13:22.695 INFO:teuthology.orchestra.console.smithi204:Power off completed 2024-03-26T04:13:22.799 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} duration: 709.758748292923 failure_reason: '"2024-03-26T04:09:07.276203+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-26T04:13:22.799 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T04:13:22.876 INFO:teuthology.run:FAIL