2024-03-12T02:39:27.408 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-12T02:39:27.408 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827 --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/orig.config.yaml 2024-03-12T02:39:27.436 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T02:39:27.517 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827 branch: main description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591827' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPae1lA8hOXE9C3ebz8WdBAdUiq63/PL99WIrU5rSMdQ/IozNnbs0Is0k8+fjt2DVuUjL9Ucqewi0+ybs21i89M= smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG4exXonK73KnghevnJSUvMY6uidkshnKuYn0e5Lr/NDha/kZVTqlucLpvoLCK0BZC+58sIwZBJESSz/LTtV+JI= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG1rTH5lOKw+pdfuFit8a68/ZAeNVwpS16sKS0MWrAZRw8TZHmAZlR6foYI+xTa0G9GttzDBCuEDQoVWIn2TN2o= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-12T02:39:27.517 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-12T02:39:27.518 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-12T02:39:27.518 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-12T02:39:27.520 INFO:teuthology.task.internal:Checking packages... 2024-03-12T02:39:27.545 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-12T02:39:27.545 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T02:39:27.545 INFO:teuthology.packaging:ref: None 2024-03-12T02:39:27.545 INFO:teuthology.packaging:tag: None 2024-03-12T02:39:27.545 INFO:teuthology.packaging:branch: main 2024-03-12T02:39:27.545 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T02:39:27.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-12T02:39:27.767 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1998.g1c349451 2024-03-12T02:39:27.769 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-12T02:39:27.775 INFO:teuthology.task.internal:no buildpackages task found 2024-03-12T02:39:27.775 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-12T02:39:27.826 INFO:teuthology.task.internal:Saving configuration 2024-03-12T02:39:27.838 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-12T02:39:27.851 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T02:39:27.873 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:34:21.062167', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPae1lA8hOXE9C3ebz8WdBAdUiq63/PL99WIrU5rSMdQ/IozNnbs0Is0k8+fjt2DVuUjL9Ucqewi0+ybs21i89M='} 2024-03-12T02:39:27.897 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:34:21.064070', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG4exXonK73KnghevnJSUvMY6uidkshnKuYn0e5Lr/NDha/kZVTqlucLpvoLCK0BZC+58sIwZBJESSz/LTtV+JI='} 2024-03-12T02:39:27.919 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:34:21.065330', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG1rTH5lOKw+pdfuFit8a68/ZAeNVwpS16sKS0MWrAZRw8TZHmAZlR6foYI+xTa0G9GttzDBCuEDQoVWIn2TN2o='} 2024-03-12T02:39:27.920 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-12T02:39:27.926 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-12T02:39:27.927 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-12T02:39:27.927 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-12T02:39:27.927 INFO:teuthology.run_tasks:Running task console_log... 2024-03-12T02:39:28.046 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe04ef2bf70>, signals=[15]) 2024-03-12T02:39:28.046 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-12T02:39:28.052 INFO:teuthology.task.internal:Opening connections... 2024-03-12T02:39:28.052 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-03-12T02:39:28.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:39:28.146 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2024-03-12T02:39:28.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:39:28.237 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-03-12T02:39:28.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:39:28.326 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-12T02:39:28.332 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-03-12T02:39:28.350 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-03-12T02:39:28.350 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-03-12T02:39:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-03-12T02:39:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-03-12T02:39:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-03-12T02:39:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-03-12T02:39:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-03-12T02:39:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-03-12T02:39:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T02:39:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-03-12T02:39:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-03-12T02:39:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T02:39:28.408 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-03-12T02:39:28.408 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T02:39:28.408 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T02:39:28.408 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T02:39:28.408 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-03-12T02:39:28.429 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2024-03-12T02:39:28.446 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2024-03-12T02:39:28.447 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2024-03-12T02:39:28.503 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2024-03-12T02:39:28.503 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="9" 2024-03-12T02:39:28.503 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="9" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el9" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:LOGO="fedora-logo-icon" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T02:39:28.504 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T02:39:28.504 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2024-03-12T02:39:28.529 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-03-12T02:39:28.546 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-03-12T02:39:28.546 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-03-12T02:39:28.604 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T02:39:28.605 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T02:39:28.605 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T02:39:28.605 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-03-12T02:39:28.629 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-12T02:39:28.637 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-12T02:39:28.701 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-12T02:39:28.702 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-03-12T02:39:28.704 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2024-03-12T02:39:28.706 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-03-12T02:39:28.722 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-12T02:39:28.728 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-12T02:39:28.728 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-03-12T02:39:28.761 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2024-03-12T02:39:28.763 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-03-12T02:39:28.785 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-12T02:39:28.864 INFO:teuthology.run_tasks:Running task kernel... 2024-03-12T02:39:28.879 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-12T02:39:28.879 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-12T02:39:28.880 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T02:39:28.880 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T02:39:28.881 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T02:39:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:39:28.897 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-12T02:39:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:39:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:39:28.899 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-03-12T02:39:28.900 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-03-12T02:39:28.954 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-361.el9.x86_64 2024-03-12T02:39:28.954 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-361.el9.x86_64 2024-03-12T02:39:28.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-03-12T02:39:28.956 INFO:teuthology.orchestra.run.smithi115.stdout:5.14.0-361.el9.x86_64 2024-03-12T02:39:28.957 INFO:teuthology.task.kernel:Running kernel on smithi115: 5.14.0-361.el9.x86_64 2024-03-12T02:39:28.957 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2024-03-12T02:39:28.958 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-361.el9.x86_64 2024-03-12T02:39:28.958 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-361.el9.x86_64 2024-03-12T02:39:28.958 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-03-12T02:39:31.102 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-12T02:39:31.136 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-12T02:39:31.186 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-12T02:39:32.560 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-12T02:39:32.778 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-12T02:39:32.812 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-12T02:39:34.436 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-03-12T02:39:36.310 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-12T02:39:36.643 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-12T02:39:38.677 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-12T02:39:40.379 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-12T02:39:41.878 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 15 kB 00:00 2024-03-12T02:39:42.786 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-12T02:39:43.088 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-12T02:39:44.345 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-03-12T02:39:45.550 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 662 kB/s | 8.0 MB 00:12 2024-03-12T02:39:46.297 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-12T02:39:48.037 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-12T02:39:48.070 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-12T02:39:50.202 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T02:39:50.203 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T02:39:50.301 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-12T02:39:50.318 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-12T02:39:50.318 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-03-12T02:39:50.318 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-12T02:39:50.318 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-12T02:39:50.318 INFO:teuthology.orchestra.run.smithi115.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T02:39:50.318 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-12T02:39:50.318 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T02:39:50.318 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T02:39:50.318 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T02:39:50.318 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:39:50.319 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-12T02:39:50.319 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-12T02:39:50.319 INFO:teuthology.orchestra.run.smithi115.stdout:Install 4 Packages 2024-03-12T02:39:50.319 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:39:50.319 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 100 M 2024-03-12T02:39:50.319 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 123 M 2024-03-12T02:39:50.319 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-12T02:39:51.204 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-12T02:39:53.395 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T02:39:53.395 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T02:39:53.537 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-12T02:39:53.546 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-12T02:39:53.546 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-12T02:39:53.546 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-12T02:39:53.546 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-12T02:39:53.546 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T02:39:53.546 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-12T02:39:53.546 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T02:39:53.546 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T02:39:53.546 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T02:39:53.547 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:39:53.547 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-12T02:39:53.547 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-12T02:39:53.547 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-03-12T02:39:53.547 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:39:53.547 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 100 M 2024-03-12T02:39:53.547 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 123 M 2024-03-12T02:39:53.548 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-12T02:39:54.001 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-03-12T02:39:56.334 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2024-03-12T02:39:57.469 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-12T02:39:59.595 INFO:teuthology.orchestra.run.smithi115.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 671 kB/s | 6.0 MB 00:09 2024-03-12T02:40:02.335 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-12T02:40:04.609 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T02:40:04.609 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T02:40:04.705 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-12T02:40:04.714 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-12T02:40:04.714 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-03-12T02:40:04.714 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-03-12T02:40:04.715 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:40:04.716 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 100 M 2024-03-12T02:40:04.716 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 123 M 2024-03-12T02:40:04.716 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-12T02:40:05.762 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 6.0 MB/s | 6.0 MB 00:01 2024-03-12T02:40:07.978 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 12 MB/s | 39 MB 00:03 2024-03-12T02:40:08.453 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 5.6 MB/s | 21 MB 00:03 2024-03-12T02:40:08.970 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 10 MB/s | 33 MB 00:03 2024-03-12T02:40:08.970 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-12T02:40:08.970 INFO:teuthology.orchestra.run.smithi079.stdout:Total 23 MB/s | 100 MB 00:04 2024-03-12T02:40:09.851 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-12T02:40:10.246 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-12T02:40:10.246 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-12T02:40:11.165 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 349 kB/s | 6.0 MB 00:17 2024-03-12T02:40:11.863 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-12T02:40:11.863 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-12T02:40:14.581 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-12T02:40:15.241 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T02:40:15.370 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:40:18.171 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:40:18.354 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:40:21.158 INFO:teuthology.orchestra.run.smithi115.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 695 kB/s | 21 MB 00:30 2024-03-12T02:40:23.091 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:40:23.221 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:40:28.000 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:40:48.284 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 390 kB/s | 21 MB 00:54 2024-03-12T02:40:52.069 INFO:teuthology.orchestra.run.smithi115.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 652 kB/s | 33 MB 00:52 2024-03-12T02:40:54.702 INFO:teuthology.orchestra.run.smithi115.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 626 kB/s | 39 MB 01:04 2024-03-12T02:40:54.702 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-12T02:40:54.702 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.5 MB/s | 100 MB 01:04 2024-03-12T02:40:55.586 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-12T02:40:55.968 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-12T02:40:55.968 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-12T02:40:57.321 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-12T02:40:57.322 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-12T02:40:57.397 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:40:57.397 INFO:teuthology.orchestra.run.smithi079.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T02:40:57.397 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:40:57.483 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:40:57.945 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:40:57.945 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T02:40:57.945 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:40:57.945 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:40:58.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:40:58.359 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:40:58.359 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-12T02:40:58.359 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T02:40:58.359 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T02:40:58.359 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T02:40:58.359 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T02:40:58.359 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:40:58.359 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-12T02:40:58.885 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T02:40:59.023 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T02:40:59.023 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T02:40:59.023 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T02:40:59.024 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T02:40:59.024 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T02:40:59.024 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-12T02:40:59.024 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T02:40:59.024 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T02:40:59.145 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T02:40:59.146 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T02:40:59.146 DEBUG:teuthology.orchestra.run.smithi079:> sudo rpm -qi grub2-tools 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:Name : grub2-tools 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:Epoch : 1 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:Version : 2.06 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:Release : 68.el9 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: x86_64 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:Group : Unspecified 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:Size : 8264254 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:License : GPLv3+ 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T02:40:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:Packager : builder@centos.org 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:Vendor : CentOS 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:Summary : Support tools for GRUB. 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:Description : 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:hardware devices. 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:40:59.187 INFO:teuthology.orchestra.run.smithi079.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T02:40:59.188 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T02:40:59.188 INFO:teuthology.task.kernel:Updating grub on smithi079 to boot 5.14.0-427.el9.x86_64 2024-03-12T02:40:59.188 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T02:40:59.843 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-03-12T02:41:00.030 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-12T02:41:00.682 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T02:41:00.827 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:41:01.227 INFO:teuthology.orchestra.run.smithi079.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T02:41:01.262 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-03-12T02:41:01.264 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-03-12T02:41:01.279 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.TzBTLUxnMu 2024-03-12T02:41:01.279 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TzBTLUxnMu 2024-03-12T02:41:01.348 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.TzBTLUxnMu 2024-03-12T02:41:01.487 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.TzBTLUxnMu is 6KB 2024-03-12T02:41:01.536 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.TzBTLUxnMu 2024-03-12T02:41:01.550 DEBUG:teuthology.orchestra.run.smithi079:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T02:41:01.615 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T02:41:01.615 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T02:41:01.615 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T02:41:01.615 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T02:41:01.616 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T02:41:01.757 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-03-12T02:41:03.597 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:41:03.770 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:41:08.481 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:41:08.681 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:41:13.410 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:41:23.247 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 475 kB/s | 33 MB 01:12 2024-03-12T02:41:31.759 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-12T02:41:31.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-12T02:41:31.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:41:38.920 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 382 kB/s | 39 MB 01:45 2024-03-12T02:41:38.921 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-12T02:41:38.921 INFO:teuthology.orchestra.run.smithi129.stdout:Total 967 kB/s | 100 MB 01:45 2024-03-12T02:41:39.793 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-12T02:41:40.165 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-12T02:41:40.165 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-12T02:41:41.816 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-12T02:41:41.816 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-12T02:41:42.542 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:41:42.542 INFO:teuthology.orchestra.run.smithi115.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T02:41:42.542 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:41:42.625 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:41:43.145 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:41:43.146 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T02:41:43.146 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:41:43.146 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:41:43.585 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:41:43.585 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:41:43.585 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-12T02:41:43.585 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T02:41:43.586 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T02:41:43.586 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T02:41:43.586 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T02:41:43.586 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:41:43.586 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-12T02:41:44.078 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T02:41:44.263 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T02:41:44.263 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T02:41:44.263 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T02:41:44.263 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T02:41:44.263 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T02:41:44.263 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-12T02:41:44.263 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T02:41:44.263 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T02:41:44.385 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T02:41:44.385 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T02:41:44.386 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -qi grub2-tools 2024-03-12T02:41:44.420 INFO:teuthology.orchestra.run.smithi115.stdout:Name : grub2-tools 2024-03-12T02:41:44.420 INFO:teuthology.orchestra.run.smithi115.stdout:Epoch : 1 2024-03-12T02:41:44.420 INFO:teuthology.orchestra.run.smithi115.stdout:Version : 2.06 2024-03-12T02:41:44.420 INFO:teuthology.orchestra.run.smithi115.stdout:Release : 68.el9 2024-03-12T02:41:44.420 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: x86_64 2024-03-12T02:41:44.420 INFO:teuthology.orchestra.run.smithi115.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T02:41:44.420 INFO:teuthology.orchestra.run.smithi115.stdout:Group : Unspecified 2024-03-12T02:41:44.420 INFO:teuthology.orchestra.run.smithi115.stdout:Size : 8264254 2024-03-12T02:41:44.420 INFO:teuthology.orchestra.run.smithi115.stdout:License : GPLv3+ 2024-03-12T02:41:44.420 INFO:teuthology.orchestra.run.smithi115.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:Packager : builder@centos.org 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:Vendor : CentOS 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:Summary : Support tools for GRUB. 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:Description : 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:hardware devices. 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:41:44.421 INFO:teuthology.orchestra.run.smithi115.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T02:41:44.422 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T02:41:44.422 INFO:teuthology.task.kernel:Updating grub on smithi115 to boot 5.14.0-427.el9.x86_64 2024-03-12T02:41:44.422 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T02:41:44.538 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-12T02:41:45.044 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2024-03-12T02:41:45.195 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T02:41:45.320 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:41:46.394 INFO:teuthology.orchestra.run.smithi115.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T02:41:46.431 INFO:teuthology.orchestra.run.smithi115.stderr:done 2024-03-12T02:41:46.434 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2024-03-12T02:41:46.449 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.uOvtZaDThf 2024-03-12T02:41:46.449 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uOvtZaDThf 2024-03-12T02:41:46.518 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.uOvtZaDThf 2024-03-12T02:41:46.661 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.uOvtZaDThf is 6KB 2024-03-12T02:41:46.709 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.uOvtZaDThf 2024-03-12T02:41:46.723 DEBUG:teuthology.orchestra.run.smithi115:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T02:41:46.790 INFO:teuthology.orchestra.run.smithi115.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T02:41:46.790 INFO:teuthology.orchestra.run.smithi115.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T02:41:46.790 INFO:teuthology.orchestra.run.smithi115.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T02:41:46.790 INFO:teuthology.orchestra.run.smithi115.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T02:41:46.792 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T02:41:46.949 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2024-03-12T02:41:48.099 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:41:48.279 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:41:50.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-03-12T02:41:53.021 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:41:53.169 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:41:57.925 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:41:59.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-12T02:41:59.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:42:02.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-03-12T02:42:14.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-12T02:42:14.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:42:16.952 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-12T02:42:16.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-12T02:42:16.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:42:27.235 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:42:27.235 INFO:teuthology.orchestra.run.smithi129.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T02:42:27.235 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:42:27.318 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:42:27.842 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:42:27.842 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T02:42:27.842 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T02:42:27.842 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T02:42:28.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T02:42:28.257 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:42:28.258 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-12T02:42:28.258 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T02:42:28.258 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T02:42:28.258 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T02:42:28.258 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T02:42:28.258 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:42:28.258 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-12T02:42:28.755 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T02:42:28.895 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T02:42:28.895 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T02:42:28.895 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T02:42:28.895 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T02:42:28.895 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T02:42:28.895 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-12T02:42:28.895 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T02:42:28.895 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T02:42:29.016 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T02:42:29.016 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T02:42:29.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-03-12T02:42:29.050 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-03-12T02:42:29.050 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-03-12T02:42:29.050 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-03-12T02:42:29.050 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 68.el9 2024-03-12T02:42:29.050 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-03-12T02:42:29.050 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T02:42:29.050 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-03-12T02:42:29.050 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8264254 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T02:42:29.051 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-03-12T02:42:29.052 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:42:29.052 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T02:42:29.052 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T02:42:29.052 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-427.el9.x86_64 2024-03-12T02:42:29.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T02:42:29.665 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-03-12T02:42:31.184 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T02:42:31.214 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-03-12T02:42:31.216 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-03-12T02:42:31.272 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.1rYGIdfJC2 2024-03-12T02:42:31.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1rYGIdfJC2 2024-03-12T02:42:31.341 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.1rYGIdfJC2 2024-03-12T02:42:31.484 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.1rYGIdfJC2 is 6KB 2024-03-12T02:42:31.532 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.1rYGIdfJC2 2024-03-12T02:42:31.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T02:42:31.614 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T02:42:31.614 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T02:42:31.615 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T02:42:31.615 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T02:42:31.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T02:42:31.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-03-12T02:42:32.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-03-12T02:42:35.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-03-12T02:42:44.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-12T02:42:44.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:42:47.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-03-12T02:42:47.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-12T02:42:47.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:42:48.030 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-03-12T02:42:48.422 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-12T02:42:48.422 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-12T02:42:48.423 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-03-12T02:42:48.437 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-427.el9.x86_64 2024-03-12T02:42:48.437 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T02:42:48.437 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T02:42:48.437 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T02:42:49.438 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-12T02:42:49.439 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T02:42:49.524 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-03-12T02:42:49.559 DEBUG:teuthology.parallel:result is None 2024-03-12T02:42:59.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-12T02:42:59.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:43:01.765 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-12T02:43:01.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-12T02:43:01.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:43:17.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-03-12T02:43:20.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-12T02:43:29.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-12T02:43:29.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:43:32.290 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-12T02:43:32.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-12T02:43:32.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:43:33.212 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-03-12T02:43:33.646 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-12T02:43:33.646 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-12T02:43:33.646 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-03-12T02:43:33.660 INFO:teuthology.orchestra.run.smithi115.stdout:5.14.0-427.el9.x86_64 2024-03-12T02:43:33.660 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T02:43:33.660 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T02:43:33.660 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T02:43:34.661 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-12T02:43:34.661 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T02:43:34.772 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2024-03-12T02:43:34.815 DEBUG:teuthology.parallel:result is None 2024-03-12T02:43:44.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-12T02:43:44.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:44:16.138 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-12T02:44:16.719 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-12T02:44:16.719 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-12T02:44:16.719 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-12T02:44:16.744 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-427.el9.x86_64 2024-03-12T02:44:16.744 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T02:44:16.744 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T02:44:16.744 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T02:44:17.744 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-12T02:44:17.745 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T02:44:17.800 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-03-12T02:44:17.928 DEBUG:teuthology.parallel:result is None 2024-03-12T02:44:17.928 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-12T02:44:17.936 INFO:teuthology.task.internal:Creating test directory... 2024-03-12T02:44:17.936 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T02:44:17.939 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T02:44:17.942 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T02:44:17.958 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-12T02:44:17.992 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-12T02:44:18.042 INFO:teuthology.task.internal:Creating archive directory... 2024-03-12T02:44:18.043 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T02:44:18.045 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T02:44:18.047 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T02:44:18.084 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-12T02:44:18.222 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-12T02:44:18.222 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T02:44:18.224 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T02:44:18.227 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T02:44:18.261 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T02:44:18.265 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T02:44:18.265 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T02:44:18.275 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T02:44:18.278 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T02:44:18.279 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T02:44:18.281 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-12T02:44:18.287 INFO:teuthology.task.internal:Configuring sudo... 2024-03-12T02:44:18.287 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T02:44:18.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T02:44:18.324 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T02:44:18.352 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-12T02:44:18.360 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-12T02:44:18.360 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T02:44:18.390 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T02:44:18.394 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T02:44:18.409 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T02:44:18.490 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T02:44:18.537 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T02:44:18.619 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T02:44:18.663 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:44:18.663 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T02:44:18.732 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T02:44:18.777 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T02:44:18.839 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T02:44:18.882 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T02:44:18.926 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:44:18.926 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T02:44:18.995 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T02:44:19.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T02:44:19.089 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T02:44:19.173 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T02:44:19.216 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:44:19.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T02:44:19.286 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-03-12T02:44:19.288 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2024-03-12T02:44:19.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-03-12T02:44:19.332 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T02:44:19.349 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T02:44:19.371 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T02:44:19.846 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-12T02:44:19.854 INFO:teuthology.task.internal:Starting timer... 2024-03-12T02:44:19.854 INFO:teuthology.run_tasks:Running task pcp... 2024-03-12T02:44:19.905 INFO:teuthology.run_tasks:Running task selinux... 2024-03-12T02:44:19.953 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-12T02:44:19.954 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-03-12T02:44:20.035 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-03-12T02:44:20.037 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2024-03-12T02:44:20.079 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2024-03-12T02:44:20.082 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-03-12T02:44:20.151 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-03-12T02:44:20.153 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-12T02:44:20.153 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-03-12T02:44:20.179 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-03-12T02:44:20.179 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2024-03-12T02:44:20.205 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2024-03-12T02:44:20.205 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-03-12T02:44:20.262 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-03-12T02:44:20.262 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi115.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi129.front.sepia.ceph.com': 'permissive'} 2024-03-12T02:44:20.263 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T02:44:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:44:20.293 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T02:44:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:44:20.324 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T02:44:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:44:20.356 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-12T02:44:20.356 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-03-12T02:44:20.396 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2024-03-12T02:44:20.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-03-12T02:44:20.472 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-12T02:44:20.481 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-12T02:44:20.486 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-12T02:44:20.604 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-12T02:44:20.621 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-12T02:44:20.622 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi115.front.sepia.ceph.com,smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-12T02:48:33.464 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi115.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-03-12T02:48:33.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-12T02:48:33.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:48:33.557 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-03-12T02:48:33.632 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-12T02:48:33.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-12T02:48:33.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:48:33.728 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-03-12T02:48:33.804 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-12T02:48:33.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-12T02:48:33.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T02:48:33.894 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-12T02:48:33.970 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-12T02:48:33.970 INFO:teuthology.run_tasks:Running task clock... 2024-03-12T02:48:33.980 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-12T02:48:33.981 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T02:48:33.981 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T02:48:33.984 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T02:48:33.984 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T02:48:33.986 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T02:48:33.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T02:48:34.012 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T02:48:34.016 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T02:48:34.028 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T02:48:34.034 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T02:48:34.042 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T02:48:34.057 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-03-12T02:48:34.059 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T02:48:34.062 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2024-03-12T02:48:34.070 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-03-12T02:48:34.076 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2024-03-12T02:48:34.087 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T02:48:34.089 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-03-12T02:48:34.092 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T02:48:34.103 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-03-12T02:48:34.103 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T02:48:34.108 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T02:48:34.119 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T02:48:34.135 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T02:48:34.155 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-03-12T02:48:34.159 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T02:48:34.159 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-03-12T02:48:34.159 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.159 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.159 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.159 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.159 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-03-12T02:48:34.163 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T02:48:34.163 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-03-12T02:48:34.163 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.163 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.163 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.163 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.184 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-12T02:48:34.189 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T02:48:34.189 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-12T02:48:34.189 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.189 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.189 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.189 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:48:34.190 INFO:teuthology.run_tasks:Running task pexec... 2024-03-12T02:48:34.201 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-12T02:48:34.201 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi079.front.sepia.ceph.com 2024-03-12T02:48:34.202 DEBUG:teuthology.orchestra.run.smithi079:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-12T02:48:34.202 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi115.front.sepia.ceph.com 2024-03-12T02:48:34.202 DEBUG:teuthology.orchestra.run.smithi115:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-12T02:48:34.202 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi129.front.sepia.ceph.com 2024-03-12T02:48:34.203 DEBUG:teuthology.orchestra.run.smithi129:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-12T02:48:34.783 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:49 ago on Tue 12 Mar 2024 02:47:45 AM UTC. 2024-03-12T02:48:34.797 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:49 ago on Tue 12 Mar 2024 02:47:45 AM UTC. 2024-03-12T02:48:34.813 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:49 ago on Tue 12 Mar 2024 02:47:45 AM UTC. 2024-03-12T02:48:34.928 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-12T02:48:34.929 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-12T02:48:34.929 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-03-12T02:48:34.929 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-12T02:48:34.929 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-12T02:48:34.929 INFO:teuthology.orchestra.run.smithi079.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-12T02:48:34.929 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:48:34.929 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-12T02:48:34.929 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-12T02:48:34.929 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-12T02:48:34.930 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:48:34.941 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.1 M 2024-03-12T02:48:34.942 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 9.8 M 2024-03-12T02:48:34.942 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-12T02:48:34.942 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-12T02:48:34.943 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-12T02:48:34.943 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-03-12T02:48:34.943 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-12T02:48:34.943 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-12T02:48:34.943 INFO:teuthology.orchestra.run.smithi115.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-12T02:48:34.943 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:48:34.943 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-12T02:48:34.944 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-12T02:48:34.944 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-12T02:48:34.944 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:48:34.959 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 3.1 M 2024-03-12T02:48:34.959 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 9.8 M 2024-03-12T02:48:34.959 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-12T02:48:34.960 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-12T02:48:34.960 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-12T02:48:34.960 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-12T02:48:34.960 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-12T02:48:34.960 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-12T02:48:34.960 INFO:teuthology.orchestra.run.smithi129.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-12T02:48:34.960 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:48:34.960 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-12T02:48:34.961 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-12T02:48:34.961 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-12T02:48:34.961 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:48:34.976 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 3.1 M 2024-03-12T02:48:34.976 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 9.8 M 2024-03-12T02:48:34.976 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-12T02:48:35.373 INFO:teuthology.orchestra.run.smithi129.stdout:runc-1.1.12-2.el9.x86_64.rpm 9.5 MB/s | 3.1 MB 00:00 2024-03-12T02:48:35.373 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-12T02:48:35.374 INFO:teuthology.orchestra.run.smithi129.stdout:Total 7.9 MB/s | 3.1 MB 00:00 2024-03-12T02:48:35.409 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-12T02:48:35.415 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-12T02:48:35.415 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-12T02:48:35.493 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-12T02:48:35.493 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-12T02:48:35.722 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-12T02:48:35.818 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T02:48:36.264 INFO:teuthology.orchestra.run.smithi079.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.0 MB/s | 3.1 MB 00:01 2024-03-12T02:48:36.264 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-12T02:48:36.264 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.4 MB/s | 3.1 MB 00:01 2024-03-12T02:48:36.301 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-12T02:48:36.307 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-12T02:48:36.307 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-12T02:48:36.386 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-12T02:48:36.386 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-12T02:48:36.394 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T02:48:36.601 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-12T02:48:36.680 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T02:48:36.680 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:48:36.680 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-12T02:48:36.680 INFO:teuthology.orchestra.run.smithi129.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-12T02:48:36.681 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:48:36.681 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-12T02:48:36.691 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T02:48:36.907 DEBUG:teuthology.parallel:result is None 2024-03-12T02:48:37.210 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T02:48:37.531 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T02:48:37.531 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:48:37.531 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-12T02:48:37.531 INFO:teuthology.orchestra.run.smithi079.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-12T02:48:37.531 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:48:37.531 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-12T02:48:37.811 DEBUG:teuthology.parallel:result is None 2024-03-12T02:48:38.548 INFO:teuthology.orchestra.run.smithi115.stdout:runc-1.1.12-2.el9.x86_64.rpm 909 kB/s | 3.1 MB 00:03 2024-03-12T02:48:38.548 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-12T02:48:38.548 INFO:teuthology.orchestra.run.smithi115.stdout:Total 892 kB/s | 3.1 MB 00:03 2024-03-12T02:48:38.586 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-12T02:48:38.593 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-12T02:48:38.593 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-12T02:48:38.673 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-12T02:48:38.674 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-12T02:48:38.880 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-12T02:48:38.959 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T02:48:39.526 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T02:48:39.852 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T02:48:39.852 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:48:39.852 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-12T02:48:39.852 INFO:teuthology.orchestra.run.smithi115.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-12T02:48:39.852 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-12T02:48:39.852 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-12T02:48:40.114 DEBUG:teuthology.parallel:result is None 2024-03-12T02:48:40.114 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-12T02:48:40.123 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-12T02:48:40.124 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:48:40.124 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T02:48:40.142 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T02:48:40.142 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-03-12T02:48:40.202 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T02:48:40.202 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:40.202 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-12T02:48:40.203 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:40.203 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:40.203 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-12 02:48:32.423281409 +0000 2024-03-12T02:48:40.203 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-12 02:48:09.503815699 +0000 2024-03-12T02:48:40.203 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-12 02:48:09.503815699 +0000 2024-03-12T02:48:40.203 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-12 02:48:09.503815699 +0000 2024-03-12T02:48:40.203 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T02:48:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-12T02:48:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-12T02:48:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000272012 s, 1.9 MB/s 2024-03-12T02:48:40.273 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T02:48:40.332 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-03-12T02:48:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T02:48:40.390 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:40.390 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-12T02:48:40.390 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:40.390 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:40.391 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-12 02:48:32.423281409 +0000 2024-03-12T02:48:40.391 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-12 02:48:09.829808076 +0000 2024-03-12T02:48:40.391 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-12 02:48:09.829808076 +0000 2024-03-12T02:48:40.391 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-12 02:48:09.829808076 +0000 2024-03-12T02:48:40.391 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T02:48:40.459 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-12T02:48:40.459 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-12T02:48:40.460 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000202718 s, 2.5 MB/s 2024-03-12T02:48:40.461 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T02:48:40.518 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-03-12T02:48:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T02:48:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:40.575 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-12T02:48:40.575 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:40.575 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:40.575 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-12 02:48:32.423281409 +0000 2024-03-12T02:48:40.575 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-12 02:48:10.158800382 +0000 2024-03-12T02:48:40.575 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-12 02:48:10.158800382 +0000 2024-03-12T02:48:40.575 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-12 02:48:10.158800382 +0000 2024-03-12T02:48:40.576 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T02:48:40.643 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-12T02:48:40.643 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-12T02:48:40.644 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000172102 s, 3.0 MB/s 2024-03-12T02:48:40.645 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T02:48:40.701 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-03-12T02:48:40.757 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T02:48:40.757 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:40.757 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-12T02:48:40.758 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:40.758 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:40.758 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-12 02:48:32.423281409 +0000 2024-03-12T02:48:40.758 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-12 02:48:10.489792642 +0000 2024-03-12T02:48:40.758 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-12 02:48:10.489792642 +0000 2024-03-12T02:48:40.758 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-12 02:48:10.489792642 +0000 2024-03-12T02:48:40.758 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T02:48:40.824 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-12T02:48:40.824 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-12T02:48:40.824 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000232785 s, 2.2 MB/s 2024-03-12T02:48:40.825 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T02:48:40.881 DEBUG:teuthology.orchestra.run.smithi079:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-12T02:48:41.024 INFO:teuthology.orchestra.run.smithi079.stdout:loop 2024-03-12T02:48:41.025 INFO:tasks.nvme_loop:Connecting nvme_loop smithi079:/dev/vg_nvme/lv_1... 2024-03-12T02:48:41.026 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-12T02:48:41.063 INFO:teuthology.orchestra.run.smithi079.stdout:1 2024-03-12T02:48:41.100 INFO:teuthology.orchestra.run.smithi079.stdout:/dev/vg_nvme/lv_11 2024-03-12T02:48:41.136 INFO:tasks.nvme_loop:Connecting nvme_loop smithi079:/dev/vg_nvme/lv_2... 2024-03-12T02:48:41.137 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-12T02:48:41.222 INFO:teuthology.orchestra.run.smithi079.stdout:1 2024-03-12T02:48:41.261 INFO:teuthology.orchestra.run.smithi079.stdout:/dev/vg_nvme/lv_21 2024-03-12T02:48:41.297 INFO:tasks.nvme_loop:Connecting nvme_loop smithi079:/dev/vg_nvme/lv_3... 2024-03-12T02:48:41.298 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-12T02:48:41.336 INFO:teuthology.orchestra.run.smithi079.stdout:1 2024-03-12T02:48:41.374 INFO:teuthology.orchestra.run.smithi079.stdout:/dev/vg_nvme/lv_31 2024-03-12T02:48:41.411 INFO:tasks.nvme_loop:Connecting nvme_loop smithi079:/dev/vg_nvme/lv_4... 2024-03-12T02:48:41.411 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-12T02:48:41.489 INFO:teuthology.orchestra.run.smithi079.stdout:1 2024-03-12T02:48:41.532 INFO:teuthology.orchestra.run.smithi079.stdout:/dev/vg_nvme/lv_41 2024-03-12T02:48:41.562 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:48:41.562 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T02:48:41.618 DEBUG:teuthology.orchestra.run.smithi079:> sudo nvme list 2024-03-12T02:48:41.683 INFO:teuthology.orchestra.run.smithi079.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-12T02:48:41.683 INFO:teuthology.orchestra.run.smithi079.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-12T02:48:41.683 INFO:teuthology.orchestra.run.smithi079.stdout:/dev/nvme0n1 PHFT6204013G400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-12T02:48:41.683 INFO:teuthology.orchestra.run.smithi079.stdout:/dev/nvme1n1 1d4d75b9f456e2261dcf Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:41.683 INFO:teuthology.orchestra.run.smithi079.stdout:/dev/nvme2n1 24de07118c241a1889d7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:41.683 INFO:teuthology.orchestra.run.smithi079.stdout:/dev/nvme3n1 8df46f27f165619ab8c3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:41.684 INFO:teuthology.orchestra.run.smithi079.stdout:/dev/nvme4n1 b7f4da26637cfca1a901 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:41.684 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T02:48:41.684 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:48:41.685 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/scratch_devs 2024-03-12T02:48:41.751 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:48:41.751 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T02:48:41.770 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T02:48:41.771 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2024-03-12T02:48:41.828 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T02:48:41.828 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:41.828 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-12T02:48:41.829 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:41.829 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:41.829 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-12 02:48:32.479141460 +0000 2024-03-12T02:48:41.829 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-12 02:48:11.078545148 +0000 2024-03-12T02:48:41.829 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-12 02:48:11.078545148 +0000 2024-03-12T02:48:41.829 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-12 02:48:11.078545148 +0000 2024-03-12T02:48:41.829 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T02:48:41.898 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-12T02:48:41.898 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-12T02:48:41.898 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000338854 s, 1.5 MB/s 2024-03-12T02:48:41.900 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T02:48:41.957 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2024-03-12T02:48:42.012 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T02:48:42.012 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:42.012 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-12T02:48:42.012 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:42.013 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:42.013 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-12 02:48:32.479141460 +0000 2024-03-12T02:48:42.013 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-12 02:48:11.413538829 +0000 2024-03-12T02:48:42.013 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-12 02:48:11.413538829 +0000 2024-03-12T02:48:42.013 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-12 02:48:11.413538829 +0000 2024-03-12T02:48:42.013 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T02:48:42.079 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-12T02:48:42.079 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-12T02:48:42.080 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000211679 s, 2.4 MB/s 2024-03-12T02:48:42.081 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T02:48:42.137 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2024-03-12T02:48:42.191 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T02:48:42.192 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:42.192 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-12T02:48:42.192 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:42.192 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:42.192 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-12 02:48:32.479141460 +0000 2024-03-12T02:48:42.192 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-12 02:48:11.749532491 +0000 2024-03-12T02:48:42.192 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-12 02:48:11.749532491 +0000 2024-03-12T02:48:42.192 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-12 02:48:11.749532491 +0000 2024-03-12T02:48:42.192 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T02:48:42.257 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-12T02:48:42.257 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-12T02:48:42.257 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.00018335 s, 2.8 MB/s 2024-03-12T02:48:42.258 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T02:48:42.314 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2024-03-12T02:48:42.369 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T02:48:42.369 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:42.369 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-12T02:48:42.369 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:42.369 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:42.369 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-12 02:48:32.479141460 +0000 2024-03-12T02:48:42.369 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-12 02:48:12.080526247 +0000 2024-03-12T02:48:42.369 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-12 02:48:12.080526247 +0000 2024-03-12T02:48:42.370 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-12 02:48:12.080526247 +0000 2024-03-12T02:48:42.370 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T02:48:42.435 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-12T02:48:42.435 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-12T02:48:42.435 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000192719 s, 2.7 MB/s 2024-03-12T02:48:42.436 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T02:48:42.492 DEBUG:teuthology.orchestra.run.smithi115:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-12T02:48:42.632 INFO:teuthology.orchestra.run.smithi115.stdout:loop 2024-03-12T02:48:42.634 INFO:tasks.nvme_loop:Connecting nvme_loop smithi115:/dev/vg_nvme/lv_1... 2024-03-12T02:48:42.634 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-12T02:48:42.671 INFO:teuthology.orchestra.run.smithi115.stdout:1 2024-03-12T02:48:42.708 INFO:teuthology.orchestra.run.smithi115.stdout:/dev/vg_nvme/lv_11 2024-03-12T02:48:42.739 INFO:tasks.nvme_loop:Connecting nvme_loop smithi115:/dev/vg_nvme/lv_2... 2024-03-12T02:48:42.739 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-12T02:48:42.818 INFO:teuthology.orchestra.run.smithi115.stdout:1 2024-03-12T02:48:42.857 INFO:teuthology.orchestra.run.smithi115.stdout:/dev/vg_nvme/lv_21 2024-03-12T02:48:42.894 INFO:tasks.nvme_loop:Connecting nvme_loop smithi115:/dev/vg_nvme/lv_3... 2024-03-12T02:48:42.894 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-12T02:48:42.972 INFO:teuthology.orchestra.run.smithi115.stdout:1 2024-03-12T02:48:43.010 INFO:teuthology.orchestra.run.smithi115.stdout:/dev/vg_nvme/lv_31 2024-03-12T02:48:43.040 INFO:tasks.nvme_loop:Connecting nvme_loop smithi115:/dev/vg_nvme/lv_4... 2024-03-12T02:48:43.040 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-12T02:48:43.120 INFO:teuthology.orchestra.run.smithi115.stdout:1 2024-03-12T02:48:43.158 INFO:teuthology.orchestra.run.smithi115.stdout:/dev/vg_nvme/lv_41 2024-03-12T02:48:43.190 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:48:43.190 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T02:48:43.244 DEBUG:teuthology.orchestra.run.smithi115:> sudo nvme list 2024-03-12T02:48:43.311 INFO:teuthology.orchestra.run.smithi115.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-12T02:48:43.311 INFO:teuthology.orchestra.run.smithi115.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-12T02:48:43.311 INFO:teuthology.orchestra.run.smithi115.stdout:/dev/nvme0n1 PHFT620400UU400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-12T02:48:43.311 INFO:teuthology.orchestra.run.smithi115.stdout:/dev/nvme1n1 bcd78b3fdfcd2cc84856 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:43.311 INFO:teuthology.orchestra.run.smithi115.stdout:/dev/nvme2n1 0fc1458256c92e106253 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:43.311 INFO:teuthology.orchestra.run.smithi115.stdout:/dev/nvme3n1 c71e70068515d2388ab9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:43.311 INFO:teuthology.orchestra.run.smithi115.stdout:/dev/nvme4n1 4a5c9d16d79f34b54f07 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:43.312 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T02:48:43.312 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:48:43.312 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/scratch_devs 2024-03-12T02:48:43.379 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:48:43.379 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T02:48:43.398 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T02:48:43.398 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-03-12T02:48:43.455 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T02:48:43.455 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:43.455 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-12T02:48:43.455 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:43.456 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:43.456 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-12 02:48:32.447018035 +0000 2024-03-12T02:48:43.456 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-12 02:48:11.093509425 +0000 2024-03-12T02:48:43.456 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-12 02:48:11.093509425 +0000 2024-03-12T02:48:43.456 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-12 02:48:11.093509425 +0000 2024-03-12T02:48:43.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-12T02:48:43.525 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000276073 s, 1.9 MB/s 2024-03-12T02:48:43.526 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T02:48:43.585 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-03-12T02:48:43.642 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T02:48:43.642 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:43.642 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-12T02:48:43.642 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:43.642 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:43.642 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-12 02:48:32.447018035 +0000 2024-03-12T02:48:43.642 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-12 02:48:11.445501325 +0000 2024-03-12T02:48:43.642 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-12 02:48:11.445501325 +0000 2024-03-12T02:48:43.642 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-12 02:48:11.445501325 +0000 2024-03-12T02:48:43.643 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T02:48:43.710 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-12T02:48:43.710 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-12T02:48:43.711 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000176642 s, 2.9 MB/s 2024-03-12T02:48:43.711 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T02:48:43.768 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-03-12T02:48:43.824 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T02:48:43.824 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:43.825 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-12T02:48:43.825 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:43.825 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:43.825 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-12 02:48:32.447018035 +0000 2024-03-12T02:48:43.825 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-12 02:48:11.790493386 +0000 2024-03-12T02:48:43.825 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-12 02:48:11.790493386 +0000 2024-03-12T02:48:43.825 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-12 02:48:11.790493386 +0000 2024-03-12T02:48:43.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T02:48:43.891 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-12T02:48:43.892 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-12T02:48:43.892 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000176432 s, 2.9 MB/s 2024-03-12T02:48:43.893 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T02:48:43.950 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-03-12T02:48:44.006 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T02:48:44.006 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T02:48:44.006 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-12T02:48:44.006 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T02:48:44.006 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T02:48:44.006 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-12 02:48:32.447018035 +0000 2024-03-12T02:48:44.006 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-12 02:48:12.143485263 +0000 2024-03-12T02:48:44.006 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-12 02:48:12.143485263 +0000 2024-03-12T02:48:44.006 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-12 02:48:12.143485263 +0000 2024-03-12T02:48:44.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T02:48:44.074 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-12T02:48:44.074 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-12T02:48:44.074 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000184996 s, 2.8 MB/s 2024-03-12T02:48:44.075 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T02:48:44.132 DEBUG:teuthology.orchestra.run.smithi129:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-12T02:48:44.299 INFO:teuthology.orchestra.run.smithi129.stdout:loop 2024-03-12T02:48:44.301 INFO:tasks.nvme_loop:Connecting nvme_loop smithi129:/dev/vg_nvme/lv_1... 2024-03-12T02:48:44.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-12T02:48:44.338 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-03-12T02:48:44.375 INFO:teuthology.orchestra.run.smithi129.stdout:/dev/vg_nvme/lv_11 2024-03-12T02:48:44.404 INFO:tasks.nvme_loop:Connecting nvme_loop smithi129:/dev/vg_nvme/lv_2... 2024-03-12T02:48:44.405 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-12T02:48:44.491 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-03-12T02:48:44.530 INFO:teuthology.orchestra.run.smithi129.stdout:/dev/vg_nvme/lv_21 2024-03-12T02:48:44.559 INFO:tasks.nvme_loop:Connecting nvme_loop smithi129:/dev/vg_nvme/lv_3... 2024-03-12T02:48:44.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-12T02:48:44.599 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-03-12T02:48:44.637 INFO:teuthology.orchestra.run.smithi129.stdout:/dev/vg_nvme/lv_31 2024-03-12T02:48:44.667 INFO:tasks.nvme_loop:Connecting nvme_loop smithi129:/dev/vg_nvme/lv_4... 2024-03-12T02:48:44.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-12T02:48:44.746 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-03-12T02:48:44.785 INFO:teuthology.orchestra.run.smithi129.stdout:/dev/vg_nvme/lv_41 2024-03-12T02:48:44.814 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:48:44.815 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T02:48:44.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo nvme list 2024-03-12T02:48:44.941 INFO:teuthology.orchestra.run.smithi129.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-12T02:48:44.941 INFO:teuthology.orchestra.run.smithi129.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-12T02:48:44.941 INFO:teuthology.orchestra.run.smithi129.stdout:/dev/nvme0n1 CVFT623300HY400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-12T02:48:44.941 INFO:teuthology.orchestra.run.smithi129.stdout:/dev/nvme1n1 670334650a0372277aaa Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:44.942 INFO:teuthology.orchestra.run.smithi129.stdout:/dev/nvme2n1 31751faaae429dde467e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:44.942 INFO:teuthology.orchestra.run.smithi129.stdout:/dev/nvme3n1 a96cbc21f96863490278 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:44.942 INFO:teuthology.orchestra.run.smithi129.stdout:/dev/nvme4n1 83ed5980b80677a16f4e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T02:48:44.943 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T02:48:44.943 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:48:44.943 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/scratch_devs 2024-03-12T02:48:45.009 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-12T02:48:45.101 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-12T02:48:45.101 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T02:48:45.102 INFO:tasks.cephadm:Cluster fsid is 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:48:45.102 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-12T02:48:45.102 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.79', 'mon.b': '172.21.15.115', 'mon.c': '172.21.15.129'} 2024-03-12T02:48:45.102 INFO:tasks.cephadm:First mon is mon.a on smithi079 2024-03-12T02:48:45.102 INFO:tasks.cephadm:First mgr is y 2024-03-12T02:48:45.102 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-12T02:48:45.103 DEBUG:teuthology.orchestra.run.smithi079:> sudo hostname $(hostname -s) 2024-03-12T02:48:45.135 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname $(hostname -s) 2024-03-12T02:48:45.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname $(hostname -s) 2024-03-12T02:48:45.195 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-12T02:48:45.195 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T02:48:45.340 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54565/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1979-g2b0101eb', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1979.g2b0101eb'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-11 00:47:00.272739', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-12T02:48:45.482 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref main, sha1 2b0101eb74d60610d544cb8a473e23a35ac1cc0b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T02:48:45.484 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm 2024-03-12T02:48:45.484 DEBUG:teuthology.orchestra.run.smithi079:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T02:48:46.013 INFO:teuthology.orchestra.run.smithi079.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 12 02:48 /home/ubuntu/cephtest/cephadm 2024-03-12T02:48:46.013 DEBUG:teuthology.orchestra.run.smithi115:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T02:48:46.670 INFO:teuthology.orchestra.run.smithi115.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 12 02:48 /home/ubuntu/cephtest/cephadm 2024-03-12T02:48:46.670 DEBUG:teuthology.orchestra.run.smithi129:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T02:48:47.147 INFO:teuthology.orchestra.run.smithi129.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 12 02:48 /home/ubuntu/cephtest/cephadm 2024-03-12T02:48:47.147 DEBUG:teuthology.orchestra.run.smithi079:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T02:48:47.166 DEBUG:teuthology.orchestra.run.smithi115:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T02:48:47.185 DEBUG:teuthology.orchestra.run.smithi129:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T02:48:47.216 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b on all hosts... 2024-03-12T02:48:47.216 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-12T02:48:47.219 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-12T02:48:47.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-12T02:48:47.966 INFO:teuthology.orchestra.run.smithi079.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T02:48:48.101 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T02:48:48.101 INFO:teuthology.orchestra.run.smithi129.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T02:49:49.246 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-03-12T02:49:49.246 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-12T02:49:49.246 INFO:teuthology.orchestra.run.smithi079.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-12T02:49:49.246 INFO:teuthology.orchestra.run.smithi079.stdout: "repo_digests": [ 2024-03-12T02:49:49.246 INFO:teuthology.orchestra.run.smithi079.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-12T02:49:49.247 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-03-12T02:49:49.247 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-03-12T02:50:10.890 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-12T02:50:10.890 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-12T02:50:10.890 INFO:teuthology.orchestra.run.smithi115.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-12T02:50:10.890 INFO:teuthology.orchestra.run.smithi115.stdout: "repo_digests": [ 2024-03-12T02:50:10.891 INFO:teuthology.orchestra.run.smithi115.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-12T02:50:10.891 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-12T02:50:10.891 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-12T02:50:13.431 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-12T02:50:13.431 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-12T02:50:13.431 INFO:teuthology.orchestra.run.smithi129.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-12T02:50:13.431 INFO:teuthology.orchestra.run.smithi129.stdout: "repo_digests": [ 2024-03-12T02:50:13.431 INFO:teuthology.orchestra.run.smithi129.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-12T02:50:13.431 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-03-12T02:50:13.431 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-12T02:50:13.454 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph 2024-03-12T02:50:13.490 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph 2024-03-12T02:50:13.527 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph 2024-03-12T02:50:13.557 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /etc/ceph 2024-03-12T02:50:13.588 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /etc/ceph 2024-03-12T02:50:13.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /etc/ceph 2024-03-12T02:50:13.644 INFO:tasks.cephadm:Writing seed config... 2024-03-12T02:50:13.645 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-12T02:50:13.645 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-12T02:50:13.645 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-12T02:50:13.646 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-12T02:50:13.646 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-12T02:50:13.646 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-12T02:50:13.646 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-12T02:50:13.646 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-12T02:50:13.646 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-12T02:50:13.646 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-12T02:50:13.647 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:50:13.647 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-12T02:50:13.664 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0b28b95e-e01b-11ee-95c8-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-12T02:50:13.664 DEBUG:teuthology.orchestra.run.smithi079:mon.a> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.a.service 2024-03-12T02:50:13.706 DEBUG:teuthology.orchestra.run.smithi079:mgr.y> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.y.service 2024-03-12T02:50:13.749 INFO:tasks.cephadm:Bootstrapping... 2024-03-12T02:50:13.750 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b -v bootstrap --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.79 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-12T02:50:13.943 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-12T02:50:13.943 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b', '-v', 'bootstrap', '--fsid', '0b28b95e-e01b-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.79', '--skip-admin-label'] 2024-03-12T02:50:13.968 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T02:50:13.969 INFO:teuthology.orchestra.run.smithi079.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-12T02:50:13.969 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying podman|docker is present... 2024-03-12T02:50:13.992 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T02:50:13.992 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying lvm2 is present... 2024-03-12T02:50:13.992 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying time synchronization is in place... 2024-03-12T02:50:13.999 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T02:50:13.999 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T02:50:14.004 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T02:50:14.005 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-03-12T02:50:14.011 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-03-12T02:50:14.018 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-03-12T02:50:14.018 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-03-12T02:50:14.018 INFO:teuthology.orchestra.run.smithi079.stdout:Repeating the final host check... 2024-03-12T02:50:14.039 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T02:50:14.039 INFO:teuthology.orchestra.run.smithi079.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-12T02:50:14.039 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl is present 2024-03-12T02:50:14.039 INFO:teuthology.orchestra.run.smithi079.stdout:lvcreate is present 2024-03-12T02:50:14.045 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T02:50:14.045 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T02:50:14.051 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T02:50:14.051 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-03-12T02:50:14.057 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-03-12T02:50:14.063 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-03-12T02:50:14.063 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-03-12T02:50:14.063 INFO:teuthology.orchestra.run.smithi079.stdout:Host looks OK 2024-03-12T02:50:14.063 INFO:teuthology.orchestra.run.smithi079.stdout:Cluster fsid: 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:50:14.063 INFO:teuthology.orchestra.run.smithi079.stdout:Acquiring lock 140013299279232 on /run/cephadm/0b28b95e-e01b-11ee-95c8-87774f69a715.lock 2024-03-12T02:50:14.063 INFO:teuthology.orchestra.run.smithi079.stdout:Lock 140013299279232 acquired on /run/cephadm/0b28b95e-e01b-11ee-95c8-87774f69a715.lock 2024-03-12T02:50:14.063 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 3300 ... 2024-03-12T02:50:14.064 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 6789 ... 2024-03-12T02:50:14.064 INFO:teuthology.orchestra.run.smithi079.stdout:Base mon IP(s) is [172.21.15.79:3300, 172.21.15.79:6789], mon addrv is [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-03-12T02:50:14.067 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.79 metric 100 2024-03-12T02:50:14.067 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.79 metric 100 2024-03-12T02:50:14.069 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-12T02:50:14.069 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-12T02:50:14.069 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2024-03-12T02:50:14.072 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-12T02:50:14.072 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-12T02:50:14.072 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T02:50:14.072 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-12T02:50:14.072 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce87/64 scope link 2024-03-12T02:50:14.072 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T02:50:14.073 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-03-12T02:50:14.073 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-03-12T02:50:14.073 INFO:teuthology.orchestra.run.smithi079.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-12T02:50:14.073 INFO:teuthology.orchestra.run.smithi079.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-12T02:50:14.073 INFO:teuthology.orchestra.run.smithi079.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T02:50:14.493 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-12T02:50:14.494 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T02:50:14.494 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Getting image source signatures 2024-03-12T02:50:14.494 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:4392e0adce526b0883d65bbedc13478c05c0e708702f1c7015b93fe77c162cd5 2024-03-12T02:50:14.494 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-12T02:50:14.494 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying config sha256:4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-12T02:50:14.494 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-12T02:50:17.912 INFO:teuthology.orchestra.run.smithi079.stdout:ceph: stdout ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T02:50:17.912 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph version: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T02:50:17.912 INFO:teuthology.orchestra.run.smithi079.stdout:Extracting ceph user uid/gid from container image... 2024-03-12T02:50:24.963 INFO:teuthology.orchestra.run.smithi079.stdout:stat: stdout 167 167 2024-03-12T02:50:24.963 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial keys... 2024-03-12T02:50:26.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQDywu9lw4RIEBAAFp8YwkeLwiQzFHe8RAes6Q== 2024-03-12T02:50:27.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQDzwu9lFZHHDBAANG/4iP/mv6lN6zik+vQJsw== 2024-03-12T02:50:28.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQD0wu9lVpuhBBAAfkwhaYmME+O/rXKnXlM50g== 2024-03-12T02:50:28.633 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial monmap... 2024-03-12T02:50:29.624 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T02:50:29.624 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-12T02:50:29.624 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:50:29.624 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T02:50:29.624 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool for a [v2:172.21.15.79:3300,v1:172.21.15.79:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T02:50:29.625 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = quincy 2024-03-12T02:50:29.625 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: set fsid to 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:50:29.625 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T02:50:29.625 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:50:29.625 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mon... 2024-03-12T02:50:30.713 INFO:teuthology.orchestra.run.smithi079.stdout:create mon.a on 2024-03-12T02:50:31.056 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-12T02:50:31.218 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0b28b95e-e01b-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-0b28b95e-e01b-11ee-95c8-87774f69a715.target. 2024-03-12T02:50:31.218 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-0b28b95e-e01b-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-0b28b95e-e01b-11ee-95c8-87774f69a715.target. 2024-03-12T02:50:31.438 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.a 2024-03-12T02:50:31.438 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-12T02:50:31.596 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0b28b95e-e01b-11ee-95c8-87774f69a715.target.wants/ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@.service. 2024-03-12T02:50:32.153 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T02:50:32.158 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-03-12T02:50:32.160 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T02:50:32.160 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-03-12T02:50:32.161 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-03-12T02:50:32.161 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T02:50:32.161 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon to start... 2024-03-12T02:50:32.161 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon... 2024-03-12T02:50:33.248 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:33 smithi079 ceph-mon[25307]: from='client.? 172.21.15.79:0/692069789' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T02:50:33.251 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout cluster: 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout id: 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout services: 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.573608s) 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout data: 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pgs: 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:33.252 INFO:teuthology.orchestra.run.smithi079.stdout:mon is available 2024-03-12T02:50:33.253 INFO:teuthology.orchestra.run.smithi079.stdout:Assimilating anything we can from ceph.conf... 2024-03-12T02:50:34.205 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:34 smithi079 ceph-mon[25307]: from='client.? 172.21.15.79:0/52192925' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T02:50:34.205 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:34 smithi079 ceph-mon[25307]: from='client.? 172.21.15.79:0/52192925' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T02:50:34.208 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:34.208 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T02:50:34.209 INFO:teuthology.orchestra.run.smithi079.stdout:Generating new minimal ceph.conf... 2024-03-12T02:50:35.173 INFO:teuthology.orchestra.run.smithi079.stdout:Restarting the monitor... 2024-03-12T02:50:35.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:35 smithi079 ceph-mon[25307]: from='client.? 172.21.15.79:0/2345707141' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:50:35.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:35 smithi079 systemd[1]: Stopping Ceph mon.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:50:35.756 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:35 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a[25284]: 2024-03-12T02:50:35.440+0000 7f5f3f2a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T02:50:35.756 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:35 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a[25284]: 2024-03-12T02:50:35.440+0000 7f5f3f2a7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-12T02:50:35.756 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:35 smithi079 podman[25638]: 2024-03-12 02:50:35.482169171 +0000 UTC m=+0.156586224 container died 79e52113f1ad5d8201900325c8f0fb4b8bf3a02b29ed28f515264fdb2044bab2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-12T02:50:35.757 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:35 smithi079 podman[25638]: 2024-03-12 02:50:35.754324248 +0000 UTC m=+0.428741303 container cleanup 79e52113f1ad5d8201900325c8f0fb4b8bf3a02b29ed28f515264fdb2044bab2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-03-12T02:50:35.757 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:35 smithi079 bash[25638]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a 2024-03-12T02:50:36.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 podman[25657]: 2024-03-12 02:50:36.014011591 +0000 UTC m=+0.530907845 container remove 79e52113f1ad5d8201900325c8f0fb4b8bf3a02b29ed28f515264fdb2044bab2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T02:50:36.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-12T02:50:36.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 systemd[1]: Stopped Ceph mon.a for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:50:36.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 systemd[1]: Starting Ceph mon.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:50:36.644 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 podman[25756]: 2024-03-12 02:50:36.280081007 +0000 UTC m=+0.017404539 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 podman[25756]: 2024-03-12 02:50:36.381648247 +0000 UTC m=+0.118971772 container create 527ba2953c6bfc6e5b53f6c9031890388821d1a5675abf1bf0501325a3055026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 podman[25756]: 2024-03-12 02:50:36.575917295 +0000 UTC m=+0.313240820 container init 527ba2953c6bfc6e5b53f6c9031890388821d1a5675abf1bf0501325a3055026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, ceph=True) 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 podman[25756]: 2024-03-12 02:50:36.581525641 +0000 UTC m=+0.318849167 container start 527ba2953c6bfc6e5b53f6c9031890388821d1a5675abf1bf0501325a3055026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 8 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: pidfile_write: ignore empty --pid-file 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: load: jerasure load: lrc 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: RocksDB version: 7.9.2 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Git sha 0 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: DB SUMMARY 2024-03-12T02:50:36.645 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: DB Session ID: R2W8UILFJ899GQWGGDWS 2024-03-12T02:50:36.646 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: CURRENT file: CURRENT 2024-03-12T02:50:36.646 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T02:50:36.646 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-12T02:50:36.646 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-12T02:50:36.646 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84609 ; 2024-03-12T02:50:36.646 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.error_if_exists: 0 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.create_if_missing: 0 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.paranoid_checks: 1 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.env: 0x563f05688b20 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.info_log: 0x563f0696d8e0 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.statistics: (nil) 2024-03-12T02:50:36.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.use_fsync: 0 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_log_file_size: 0 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.allow_fallocate: 1 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.use_direct_reads: 0 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.db_log_dir: 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.wal_dir: 2024-03-12T02:50:36.648 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.write_buffer_manager: 0x563f06941400 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T02:50:36.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.unordered_write: 0 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.row_cache: None 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.wal_filter: None 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.two_write_queues: 0 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T02:50:36.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.wal_compression: 0 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.atomic_flush: 0 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.log_readahead_size: 0 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_background_jobs: 2 2024-03-12T02:50:36.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_background_compactions: -1 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_subcompactions: 1 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_open_files: -1 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T02:50:36.652 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_background_flushes: -1 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Compression algorithms supported: 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: kZSTD supported: 0 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: kXpressCompression supported: 0 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: kZlibCompression supported: 1 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: kSnappyCompression supported: 1 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: kLZ4Compression supported: 1 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: kBZip2Compression supported: 0 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T02:50:36.653 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.merge_operator: 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_filter: None 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563f0696d9e0) 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout: cache_index_and_filter_blocks: 1 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T02:50:36.654 INFO:journalctl@ceph.mon.a.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: pin_top_level_index_and_filter: 1 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_type: 0 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: data_block_index_type: 0 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_shortening: 1 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: checksum: 4 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: no_block_cache: 0 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache: 0x563f06973090 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_name: BinnedLRUCache 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_options: 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: capacity : 536870912 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: num_shard_bits : 4 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: strict_capacity_limit : 0 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: high_pri_pool_ratio: 0.000 2024-03-12T02:50:36.655 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_compressed: (nil) 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: persistent_cache: (nil) 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_size: 4096 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_size_deviation: 10 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_restart_interval: 16 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_block_restart_interval: 1 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: metadata_block_size: 4096 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: partition_filters: 0 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: use_delta_encoding: 1 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: filter_policy: bloomfilter 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: whole_key_filtering: 1 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: verify_compression: 0 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: read_amp_bytes_per_bit: 0 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: format_version: 5 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: enable_index_compression: 1 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_align: 0 2024-03-12T02:50:36.656 INFO:journalctl@ceph.mon.a.smithi079.stdout: max_auto_readahead_size: 262144 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout: prepopulate_block_cache: 0 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout: initial_auto_readahead_size: 8192 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compression: NoCompression 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.num_levels: 7 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T02:50:36.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T02:50:36.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T02:50:36.659 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T02:50:36.659 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T02:50:36.659 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T02:50:36.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T02:50:36.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.table_properties_collectors: 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.inplace_update_support: 0 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.bloom_locality: 0 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.max_successive_merges: 0 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T02:50:36.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.ttl: 2592000 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.enable_blob_files: false 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.min_blob_size: 0 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-12T02:50:36.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 606a9225-9656-4411-83bc-bb65ca7efb51 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710211836610243, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710211836611084, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81234, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79421, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73730, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710211836, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "606a9225-9656-4411-83bc-bb65ca7efb51", "db_session_id": "R2W8UILFJ899GQWGGDWS", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710211836611166, "job": 1, "event": "recovery_finished"} 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563f06a66000 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: DB pointer 0x563f06a50000 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** DB Stats ** 2024-03-12T02:50:36.664 INFO:journalctl@ceph.mon.a.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** Compaction Stats [default] ** 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: L0 2/0 81.17 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: Sum 2/0 81.17 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-03-12T02:50:36.665 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** Compaction Stats [default] ** 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T02:50:36.666 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative compaction: 0.00 GB write, 27.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval compaction: 0.00 GB write, 27.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: Block cache BinnedLRUCache@0x563f06973090#8 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: starting mon.a rank 0 at public addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] at bind addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: mon.a@-1(???) e1 preinit fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: mon.a@-1(???).mds e1 new map 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: mon.a@-1(???).mds e1 print_map 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: e1 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: legacy client fscid: -1 2024-03-12T02:50:36.667 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-03-12T02:50:36.668 INFO:journalctl@ceph.mon.a.smithi079.stdout: No filesystems configured 2024-03-12T02:50:36.668 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T02:50:36.668 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T02:50:36.668 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T02:50:36.668 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T02:50:36.668 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-12T02:50:36.668 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 bash[25756]: 527ba2953c6bfc6e5b53f6c9031890388821d1a5675abf1bf0501325a3055026 2024-03-12T02:50:36.668 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 systemd[1]: Started Ceph mon.a for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:50:36.668 INFO:teuthology.orchestra.run.smithi079.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-12T02:50:36.964 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T02:50:36.964 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: monmap e1: 1 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T02:50:36.964 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: fsmap 2024-03-12T02:50:36.964 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T02:50:36.964 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:36 smithi079 ceph-mon[25794]: mgrmap e1: no daemons active 2024-03-12T02:50:37.696 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-12T02:50:37.709 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-12T02:50:37.709 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mgr... 2024-03-12T02:50:37.709 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-12T02:50:37.710 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-12T02:50:37.961 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.y 2024-03-12T02:50:37.961 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-12T02:50:38.111 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0b28b95e-e01b-11ee-95c8-87774f69a715.target.wants/ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@.service. 2024-03-12T02:50:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:38 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/3967030262' entity='client.admin' 2024-03-12T02:50:38.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:38 smithi079 podman[26036]: 2024-03-12 02:50:38.343253893 +0000 UTC m=+0.116171562 container create 13cc77d2a103ce63930492462bead608dab34f9f1da5e41d54822cf30750de7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS) 2024-03-12T02:50:38.660 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T02:50:38.660 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-03-12T02:50:38.667 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T02:50:38.667 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-03-12T02:50:38.667 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-03-12T02:50:38.668 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T02:50:38.675 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T02:50:38.675 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-03-12T02:50:38.682 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T02:50:38.682 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-03-12T02:50:38.683 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-03-12T02:50:38.683 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-12T02:50:38.683 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr to start... 2024-03-12T02:50:38.683 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr... 2024-03-12T02:50:38.808 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:38 smithi079 podman[26036]: 2024-03-12 02:50:38.537618043 +0000 UTC m=+0.310535712 container init 13cc77d2a103ce63930492462bead608dab34f9f1da5e41d54822cf30750de7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-12T02:50:38.808 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:38 smithi079 podman[26036]: 2024-03-12 02:50:38.542752307 +0000 UTC m=+0.315669977 container start 13cc77d2a103ce63930492462bead608dab34f9f1da5e41d54822cf30750de7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y, RELEASE=main-2b0101e, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T02:50:38.808 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:38 smithi079 bash[26036]: 13cc77d2a103ce63930492462bead608dab34f9f1da5e41d54822cf30750de7c 2024-03-12T02:50:38.808 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:38 smithi079 systemd[1]: Started Ceph mgr.y for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:50:39.346 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:39 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:39.092+0000 7fc9d1ae7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T02:50:39.346 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:39 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:39.188+0000 7fc9d1ae7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T02:50:39.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:39 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/3830898349' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T02:50:39.949 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:39 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:39.855+0000 7fc9d1ae7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T02:50:39.949 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:39 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:39.946+0000 7fc9d1ae7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "0b28b95e-e01b-11ee-95c8-87774f69a715", 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T02:50:39.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T02:50:39.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T02:50:39.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T02:50:39.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T02:50:39.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T02:50:39.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T02:50:39.974 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:39.974 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T02:50:39.974 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T02:50:39.974 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T02:50:39.974 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T02:50:39.974 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T02:50:39.974 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T02:50:39.974 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T02:50:39.974 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T02:50:39.975 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:39.976 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T02:50:39.976 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:39.976 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T02:50:39.976 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:39.976 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T02:50:32.098898+0000", 2024-03-12T02:50:39.976 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T02:50:39.976 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:39.976 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T02:50:39.976 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-03-12T02:50:39.976 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (1/15)... 2024-03-12T02:50:40.265 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:40 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:40.087+0000 7fc9d1ae7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T02:50:40.265 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:40 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:40.181+0000 7fc9d1ae7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T02:50:40.265 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:40 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:40.262+0000 7fc9d1ae7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T02:50:40.809 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:40 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:40.529+0000 7fc9d1ae7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T02:50:40.809 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:40 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:40.638+0000 7fc9d1ae7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T02:50:40.810 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:40 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:40.725+0000 7fc9d1ae7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T02:50:41.309 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:40 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:40.820+0000 7fc9d1ae7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T02:50:41.309 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:40 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:40.993+0000 7fc9d1ae7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T02:50:42.201 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:41 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:41.945+0000 7fc9d1ae7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T02:50:42.201 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:42 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:42.078+0000 7fc9d1ae7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T02:50:42.531 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:42 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:42.416+0000 7fc9d1ae7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T02:50:42.531 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:42 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:42.528+0000 7fc9d1ae7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T02:50:42.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:42 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2035948597' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T02:50:42.809 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:42 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:42.726+0000 7fc9d1ae7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:50:43.120 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:42 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:42.912+0000 7fc9d1ae7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:43.128 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "0b28b95e-e01b-11ee-95c8-87774f69a715", 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-12T02:50:43.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:43.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T02:50:43.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T02:50:32.098898+0000", 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T02:50:43.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-03-12T02:50:43.133 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (2/15)... 2024-03-12T02:50:43.451 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:43 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:43.194+0000 7fc9d1ae7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:50:43.809 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:43 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:43.448+0000 7fc9d1ae7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T02:50:43.809 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:43 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:43.681+0000 7fc9d1ae7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T02:50:44.810 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:44 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:44.471+0000 7fc9d1ae7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T02:50:45.309 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:45.043+0000 7fc9d1ae7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T02:50:45.309 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:45.140+0000 7fc9d1ae7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T02:50:45.598 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:45.369+0000 7fc9d1ae7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: Activating manager daemon y 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: mgrmap e2: y(active, starting, since 0.00331007s) 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: from='mgr.14100 172.21.15.79:0/3824917899' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: from='mgr.14100 172.21.15.79:0/3824917899' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: from='mgr.14100 172.21.15.79:0/3824917899' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: from='mgr.14100 172.21.15.79:0/3824917899' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: from='mgr.14100 172.21.15.79:0/3824917899' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: Manager daemon y is now available 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: from='mgr.14100 172.21.15.79:0/3824917899' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: from='mgr.14100 172.21.15.79:0/3824917899' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: from='mgr.14100 172.21.15.79:0/3824917899' entity='mgr.y' 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: from='mgr.14100 172.21.15.79:0/3824917899' entity='mgr.y' 2024-03-12T02:50:45.599 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:45 smithi079 ceph-mon[25794]: from='mgr.14100 172.21.15.79:0/3824917899' entity='mgr.y' 2024-03-12T02:50:46.240 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:46.240 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-03-12T02:50:46.240 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "0b28b95e-e01b-11ee-95c8-87774f69a715", 2024-03-12T02:50:46.240 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T02:50:46.240 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T02:50:46.240 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:46.241 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T02:50:46.242 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:46.242 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T02:50:46.242 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T02:50:46.242 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T02:50:46.242 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T02:50:46.242 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T02:50:46.242 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T02:50:46.245 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:46.245 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T02:50:46.245 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T02:50:46.245 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T02:50:46.245 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T02:50:46.245 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T02:50:46.245 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T02:50:46.246 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T02:50:32.098898+0000", 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-03-12T02:50:46.247 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (3/15)... 2024-03-12T02:50:46.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:46 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2129184971' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T02:50:46.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:46 smithi079 ceph-mon[25794]: mgrmap e3: y(active, since 1.00672s) 2024-03-12T02:50:48.688 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:48 smithi079 ceph-mon[25794]: mgrmap e4: y(active, since 2s) 2024-03-12T02:50:49.535 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:49 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/234407285' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T02:50:49.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:49.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-03-12T02:50:49.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "0b28b95e-e01b-11ee-95c8-87774f69a715", 2024-03-12T02:50:49.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T02:50:49.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T02:50:49.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T02:50:49.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T02:50:49.564 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T02:50:49.565 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:49.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T02:50:49.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T02:50:49.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T02:50:49.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T02:50:49.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T02:50:49.566 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T02:50:49.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:49.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T02:50:49.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T02:50:49.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T02:50:49.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T02:50:49.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T02:50:49.569 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T02:50:49.570 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T02:50:32.098898+0000", 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-03-12T02:50:49.571 INFO:teuthology.orchestra.run.smithi079.stdout:mgr is available 2024-03-12T02:50:50.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:50 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/756706368' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T02:50:50.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:50 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/756706368' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T02:50:50.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-03-12T02:50:50.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T02:50:50.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T02:50:50.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T02:50:50.921 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling cephadm module... 2024-03-12T02:50:52.014 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:51 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1725546620' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-12T02:50:52.014 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:51 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ignoring --setuser ceph since I am not root 2024-03-12T02:50:52.014 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:51 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ignoring --setgroup ceph since I am not root 2024-03-12T02:50:52.559 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:52 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:52.273+0000 7f253c9ec200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T02:50:52.559 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:52 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:52.370+0000 7f253c9ec200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T02:50:52.997 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:52 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1725546620' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-12T02:50:52.997 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:52 smithi079 ceph-mon[25794]: mgrmap e5: y(active, since 6s) 2024-03-12T02:50:53.273 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:53.037+0000 7f253c9ec200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T02:50:53.273 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:53.128+0000 7f253c9ec200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T02:50:53.273 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:53.270+0000 7f253c9ec200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T02:50:53.546 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:53.366+0000 7f253c9ec200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T02:50:53.547 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:53.448+0000 7f253c9ec200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T02:50:53.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-03-12T02:50:53.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-12T02:50:53.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T02:50:53.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-12T02:50:53.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T02:50:53.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-03-12T02:50:53.551 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-03-12T02:50:53.551 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 5... 2024-03-12T02:50:53.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:53 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2603461235' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T02:50:53.809 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:53.722+0000 7f253c9ec200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T02:50:54.198 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:53.834+0000 7f253c9ec200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T02:50:54.198 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:53.922+0000 7f253c9ec200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T02:50:54.198 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:54 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:54.018+0000 7f253c9ec200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T02:50:54.559 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:54 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:54.195+0000 7f253c9ec200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T02:50:55.559 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:55 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:55.136+0000 7f253c9ec200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T02:50:55.559 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:55 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:55.264+0000 7f253c9ec200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T02:50:55.879 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:55 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:55.596+0000 7f253c9ec200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T02:50:55.879 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:55 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:55.684+0000 7f253c9ec200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T02:50:55.879 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:55 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:55.876+0000 7f253c9ec200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:50:56.309 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:56 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:56.058+0000 7f253c9ec200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T02:50:56.597 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:56 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:56.339+0000 7f253c9ec200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:50:56.597 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:56 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:56.594+0000 7f253c9ec200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T02:50:57.059 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:56 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:56.829+0000 7f253c9ec200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T02:50:58.059 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:57 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:57.622+0000 7f253c9ec200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T02:50:58.518 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:58.205+0000 7f253c9ec200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T02:50:58.518 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:58.300+0000 7f253c9ec200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T02:50:58.518 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:50:58.515+0000 7f253c9ec200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T02:50:58.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: Active manager daemon y restarted 2024-03-12T02:50:58.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: Activating manager daemon y 2024-03-12T02:50:58.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: osdmap e2: 0 total, 0 up, 0 in 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: mgrmap e6: y(active, starting, since 0.00345151s) 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: Manager daemon y is now available 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:50:58.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:58 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:50:59.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:59 smithi079 ceph-mon[25794]: Found migration_current of "None". Setting to last migration. 2024-03-12T02:50:59.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:59 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:50:59.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:59 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T02:50:59.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:59 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T02:50:59.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:59 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:50:59.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:59 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:50:59.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:50:59 smithi079 ceph-mon[25794]: mgrmap e7: y(active, since 1.0055s) 2024-03-12T02:51:00.148 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-03-12T02:51:00.148 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-12T02:51:00.148 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T02:51:00.148 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-03-12T02:51:00.148 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 5 is available 2024-03-12T02:51:00.148 INFO:teuthology.orchestra.run.smithi079.stdout:Setting orchestrator backend to cephadm... 2024-03-12T02:51:00.800 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:00 smithi079 ceph-mon[25794]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T02:51:00.800 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:00 smithi079 ceph-mon[25794]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T02:51:00.801 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:00 smithi079 ceph-mon[25794]: [12/Mar/2024:02:50:59] ENGINE Bus STARTING 2024-03-12T02:51:00.801 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:00 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:51:01.680 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:01 smithi079 ceph-mon[25794]: [12/Mar/2024:02:50:59] ENGINE Serving on https://172.21.15.79:7150 2024-03-12T02:51:01.680 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:01 smithi079 ceph-mon[25794]: [12/Mar/2024:02:50:59] ENGINE Serving on http://172.21.15.79:8765 2024-03-12T02:51:01.680 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:01 smithi079 ceph-mon[25794]: [12/Mar/2024:02:50:59] ENGINE Bus STARTED 2024-03-12T02:51:01.680 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:01 smithi079 ceph-mon[25794]: mgrmap e8: y(active, since 2s) 2024-03-12T02:51:01.680 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:01 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:51:01.680 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:01 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:51:02.528 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-12T02:51:02.528 INFO:teuthology.orchestra.run.smithi079.stdout:Generating ssh key... 2024-03-12T02:51:02.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:02 smithi079 ceph-mon[25794]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:03.881 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-mon[25794]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:03.882 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: Generating public/private rsa key pair. 2024-03-12T02:51:03.882 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: Your identification has been saved in /tmp/tmplvoyf1t4/key. 2024-03-12T02:51:03.882 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: Your public key has been saved in /tmp/tmplvoyf1t4/key.pub. 2024-03-12T02:51:03.882 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: The key fingerprint is: 2024-03-12T02:51:03.882 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: SHA256:qAnzN1fXOqRloTRurpLoyjSU4D+gKpwHFE9K6JC19Ik ceph-0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: The key's randomart image is: 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: +---[RSA 3072]----+ 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: |.oo | 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: |++ = . | 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: |= E o o . | 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: |.= o . o o o | 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: |.o= . S = = . | 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: |.oo+ o + * . | 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: |o =o+.o.. o o | 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: |o= oo.oo . . | 2024-03-12T02:51:03.883 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: |o +o. .. | 2024-03-12T02:51:03.884 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: +----[SHA256]-----+ 2024-03-12T02:51:04.803 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:04 smithi079 ceph-mon[25794]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:04.803 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:04 smithi079 ceph-mon[25794]: Generating ssh key... 2024-03-12T02:51:04.803 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:04 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:51:04.803 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:04 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:51:05.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsn6HBtunVth6g6h/031T+OePt3wxJ4wwtud1DeKFKBo9w9VhRd4keHgaYAsauPGrwkUHetay617yR8jpqp/yd/j+kMSed453l0/djiJ10wUi1L6q4dnggFZnU2TYQo7xt7Q12e6f8PlDT9Zyg0dHPAP6BgvQqDQboga4Oietu98QldfKZrgjVfcREBTmghcdSGrLBEgcUZEWUkvxYr7u1Me6tHMNyv+f6b7++jac2mDVeynyAd+MjusR44Xer3a3khpAWaVW/aSuHBDy1w/iOFXjuHubx8J4UiMnKgVjYJwZX48ok3WmbnV0cKDwJhTJ3oQMeZAT5ePMQEeq+BNpDLz7Q2VIiXBS/oaol04xaekLTr+UKW8RnmfcJWSnVwB6DE62UlWVLo6TxrDsr1lRi+LlhVH1bs2fiUxgS4daQ72DZ0ij0Q88UhB0U5skWTGpSKGk1akkpTXIl2glqIwaRddfBZ3mqtNvTF08xM6hQXFysZPXvpC+0uOvZWByecFk= ceph-0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:51:05.318 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-12T02:51:05.318 INFO:teuthology.orchestra.run.smithi079.stdout:Adding key to root@localhost authorized_keys... 2024-03-12T02:51:05.318 INFO:teuthology.orchestra.run.smithi079.stdout:Adding host smithi079... 2024-03-12T02:51:06.030 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:05 smithi079 ceph-mon[25794]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:07.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:06 smithi079 ceph-mon[25794]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:07.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:06 smithi079 ceph-mon[25794]: Deploying cephadm binary to smithi079 2024-03-12T02:51:10.800 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Added host 'smithi079' with addr '172.21.15.79' 2024-03-12T02:51:10.801 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying unmanaged mon service... 2024-03-12T02:51:11.510 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:11 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:51:11.510 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:11 smithi079 ceph-mon[25794]: Added host smithi079 2024-03-12T02:51:11.510 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:11 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:51:12.068 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-12T02:51:12.068 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying unmanaged mgr service... 2024-03-12T02:51:12.762 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:12 smithi079 ceph-mon[25794]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:12.762 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:12 smithi079 ceph-mon[25794]: Saving service mon spec with placement count:5 2024-03-12T02:51:12.762 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:12 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:51:13.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-12T02:51:13.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:13 smithi079 ceph-mon[25794]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:13.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:13 smithi079 ceph-mon[25794]: Saving service mgr spec with placement count:2 2024-03-12T02:51:13.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:13 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:51:15.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:14 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2493808862' entity='client.admin' 2024-03-12T02:51:15.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:14 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:51:16.278 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling the dashboard module... 2024-03-12T02:51:16.671 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:16 smithi079 ceph-mon[25794]: from='mgr.14120 172.21.15.79:0/2090873012' entity='mgr.y' 2024-03-12T02:51:16.672 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:16 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/488101310' entity='client.admin' 2024-03-12T02:51:17.713 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:17 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/495228466' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-12T02:51:17.714 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:17 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ignoring --setuser ceph since I am not root 2024-03-12T02:51:17.714 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:17 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ignoring --setgroup ceph since I am not root 2024-03-12T02:51:18.244 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:17 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:17.990+0000 7f2340a3b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T02:51:18.244 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:18.084+0000 7f2340a3b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T02:51:18.517 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:18 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/495228466' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-12T02:51:18.518 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:18 smithi079 ceph-mon[25794]: mgrmap e9: y(active, since 18s) 2024-03-12T02:51:18.809 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:18.750+0000 7f2340a3b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T02:51:19.174 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:18.844+0000 7f2340a3b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T02:51:19.174 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:18.992+0000 7f2340a3b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T02:51:19.174 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:19 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:19.086+0000 7f2340a3b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T02:51:19.174 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:19 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:19.171+0000 7f2340a3b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T02:51:19.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-03-12T02:51:19.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-12T02:51:19.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T02:51:19.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-12T02:51:19.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T02:51:19.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-03-12T02:51:19.457 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-03-12T02:51:19.457 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 9... 2024-03-12T02:51:19.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:19 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/851713573' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T02:51:19.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:19 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:19.438+0000 7f2340a3b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T02:51:19.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:19 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:19.552+0000 7f2340a3b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T02:51:19.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:19 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:19.640+0000 7f2340a3b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T02:51:19.948 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:19 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:19.750+0000 7f2340a3b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T02:51:19.948 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:19 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:19.933+0000 7f2340a3b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T02:51:21.309 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:20 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:20.912+0000 7f2340a3b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T02:51:21.310 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:21 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:21.042+0000 7f2340a3b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T02:51:21.658 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:21 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:21.372+0000 7f2340a3b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T02:51:21.659 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:21 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:21.461+0000 7f2340a3b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T02:51:21.659 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:21 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:21.655+0000 7f2340a3b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:51:22.059 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:21 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:21.840+0000 7f2340a3b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T02:51:22.378 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:22 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:22.119+0000 7f2340a3b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:51:22.378 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:22 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:22.375+0000 7f2340a3b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T02:51:22.809 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:22 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:22.607+0000 7f2340a3b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T02:51:23.809 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:23 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:23.392+0000 7f2340a3b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T02:51:24.272 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:23 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:23.956+0000 7f2340a3b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T02:51:24.272 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:24.055+0000 7f2340a3b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T02:51:24.272 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:51:24.269+0000 7f2340a3b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: Active manager daemon y restarted 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: Activating manager daemon y 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: osdmap e3: 0 total, 0 up, 0 in 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: mgrmap e10: y(active, starting, since 0.00322664s) 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: Manager daemon y is now available 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:51:24.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:51:25.549 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:25 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T02:51:25.549 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:25 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T02:51:25.550 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:25 smithi079 ceph-mon[25794]: mgrmap e11: y(active, since 1.0056s) 2024-03-12T02:51:26.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-03-12T02:51:26.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-12T02:51:26.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T02:51:26.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-03-12T02:51:26.192 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 9 is available 2024-03-12T02:51:26.192 INFO:teuthology.orchestra.run.smithi079.stdout:Generating a dashboard self-signed certificate... 2024-03-12T02:51:26.509 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:26 smithi079 ceph-mon[25794]: [12/Mar/2024:02:51:24] ENGINE Bus STARTING 2024-03-12T02:51:26.509 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:26 smithi079 ceph-mon[25794]: [12/Mar/2024:02:51:24] ENGINE Serving on http://172.21.15.79:8765 2024-03-12T02:51:26.509 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:26 smithi079 ceph-mon[25794]: [12/Mar/2024:02:51:24] ENGINE Serving on https://172.21.15.79:7150 2024-03-12T02:51:26.509 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:26 smithi079 ceph-mon[25794]: [12/Mar/2024:02:51:24] ENGINE Bus STARTED 2024-03-12T02:51:26.509 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:26 smithi079 ceph-mon[25794]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T02:51:26.509 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:26 smithi079 ceph-mon[25794]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T02:51:26.509 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:27.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-12T02:51:27.516 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial admin user... 2024-03-12T02:51:28.208 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:27 smithi079 ceph-mon[25794]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:28.209 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:27 smithi079 ceph-mon[25794]: mgrmap e12: y(active, since 2s) 2024-03-12T02:51:28.209 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:28.209 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:28.209 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:29.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$1W6Clj3ycaBMdnommn/5vu0iMM8u91wBLxLhH59Cn2dC3J9hZGuSy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710211888, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-12T02:51:29.302 INFO:teuthology.orchestra.run.smithi079.stdout:Fetching dashboard port number... 2024-03-12T02:51:29.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:29 smithi079 ceph-mon[25794]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:29.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:29 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:30.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 8443 2024-03-12T02:51:30.928 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T02:51:30.928 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-03-12T02:51:30.935 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T02:51:30.935 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-03-12T02:51:30.935 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-03-12T02:51:30.935 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-12T02:51:30.936 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph Dashboard is now available at: 2024-03-12T02:51:30.936 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:30.937 INFO:teuthology.orchestra.run.smithi079.stdout: URL: https://smithi079.front.sepia.ceph.com:8443/ 2024-03-12T02:51:30.937 INFO:teuthology.orchestra.run.smithi079.stdout: User: admin 2024-03-12T02:51:30.937 INFO:teuthology.orchestra.run.smithi079.stdout: Password: nptfimpw07 2024-03-12T02:51:30.937 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:30.937 INFO:teuthology.orchestra.run.smithi079.stdout:Saving cluster configuration to /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config directory 2024-03-12T02:51:30.937 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling autotune for osd_memory_target 2024-03-12T02:51:31.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:31 smithi079 ceph-mon[25794]: mgrmap e13: y(active, since 6s) 2024-03-12T02:51:31.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:31 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/396415525' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-12T02:51:33.500 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-12T02:51:33.500 INFO:teuthology.orchestra.run.smithi079.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-12T02:51:33.500 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:33.500 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-12T02:51:33.500 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:33.500 INFO:teuthology.orchestra.run.smithi079.stdout:Or, if you are only running a single cluster on this host: 2024-03-12T02:51:33.500 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout: ceph telemetry on 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout:For more information see: 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:33.501 INFO:teuthology.orchestra.run.smithi079.stdout:Bootstrap complete. 2024-03-12T02:51:33.548 INFO:tasks.cephadm:Fetching config... 2024-03-12T02:51:33.549 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:51:33.549 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-12T02:51:33.565 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-12T02:51:33.565 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:51:33.565 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-12T02:51:33.620 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-12T02:51:33.621 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:51:33.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-12T02:51:33.688 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-12T02:51:33.688 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:51:33.688 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-12T02:51:33.745 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-12T02:51:33.745 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsn6HBtunVth6g6h/031T+OePt3wxJ4wwtud1DeKFKBo9w9VhRd4keHgaYAsauPGrwkUHetay617yR8jpqp/yd/j+kMSed453l0/djiJ10wUi1L6q4dnggFZnU2TYQo7xt7Q12e6f8PlDT9Zyg0dHPAP6BgvQqDQboga4Oietu98QldfKZrgjVfcREBTmghcdSGrLBEgcUZEWUkvxYr7u1Me6tHMNyv+f6b7++jac2mDVeynyAd+MjusR44Xer3a3khpAWaVW/aSuHBDy1w/iOFXjuHubx8J4UiMnKgVjYJwZX48ok3WmbnV0cKDwJhTJ3oQMeZAT5ePMQEeq+BNpDLz7Q2VIiXBS/oaol04xaekLTr+UKW8RnmfcJWSnVwB6DE62UlWVLo6TxrDsr1lRi+LlhVH1bs2fiUxgS4daQ72DZ0ij0Q88UhB0U5skWTGpSKGk1akkpTXIl2glqIwaRddfBZ3mqtNvTF08xM6hQXFysZPXvpC+0uOvZWByecFk= ceph-0b28b95e-e01b-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T02:51:33.843 INFO:teuthology.orchestra.run.smithi079.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsn6HBtunVth6g6h/031T+OePt3wxJ4wwtud1DeKFKBo9w9VhRd4keHgaYAsauPGrwkUHetay617yR8jpqp/yd/j+kMSed453l0/djiJ10wUi1L6q4dnggFZnU2TYQo7xt7Q12e6f8PlDT9Zyg0dHPAP6BgvQqDQboga4Oietu98QldfKZrgjVfcREBTmghcdSGrLBEgcUZEWUkvxYr7u1Me6tHMNyv+f6b7++jac2mDVeynyAd+MjusR44Xer3a3khpAWaVW/aSuHBDy1w/iOFXjuHubx8J4UiMnKgVjYJwZX48ok3WmbnV0cKDwJhTJ3oQMeZAT5ePMQEeq+BNpDLz7Q2VIiXBS/oaol04xaekLTr+UKW8RnmfcJWSnVwB6DE62UlWVLo6TxrDsr1lRi+LlhVH1bs2fiUxgS4daQ72DZ0ij0Q88UhB0U5skWTGpSKGk1akkpTXIl2glqIwaRddfBZ3mqtNvTF08xM6hQXFysZPXvpC+0uOvZWByecFk= ceph-0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:51:33.858 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsn6HBtunVth6g6h/031T+OePt3wxJ4wwtud1DeKFKBo9w9VhRd4keHgaYAsauPGrwkUHetay617yR8jpqp/yd/j+kMSed453l0/djiJ10wUi1L6q4dnggFZnU2TYQo7xt7Q12e6f8PlDT9Zyg0dHPAP6BgvQqDQboga4Oietu98QldfKZrgjVfcREBTmghcdSGrLBEgcUZEWUkvxYr7u1Me6tHMNyv+f6b7++jac2mDVeynyAd+MjusR44Xer3a3khpAWaVW/aSuHBDy1w/iOFXjuHubx8J4UiMnKgVjYJwZX48ok3WmbnV0cKDwJhTJ3oQMeZAT5ePMQEeq+BNpDLz7Q2VIiXBS/oaol04xaekLTr+UKW8RnmfcJWSnVwB6DE62UlWVLo6TxrDsr1lRi+LlhVH1bs2fiUxgS4daQ72DZ0ij0Q88UhB0U5skWTGpSKGk1akkpTXIl2glqIwaRddfBZ3mqtNvTF08xM6hQXFysZPXvpC+0uOvZWByecFk= ceph-0b28b95e-e01b-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T02:51:33.905 INFO:teuthology.orchestra.run.smithi115.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsn6HBtunVth6g6h/031T+OePt3wxJ4wwtud1DeKFKBo9w9VhRd4keHgaYAsauPGrwkUHetay617yR8jpqp/yd/j+kMSed453l0/djiJ10wUi1L6q4dnggFZnU2TYQo7xt7Q12e6f8PlDT9Zyg0dHPAP6BgvQqDQboga4Oietu98QldfKZrgjVfcREBTmghcdSGrLBEgcUZEWUkvxYr7u1Me6tHMNyv+f6b7++jac2mDVeynyAd+MjusR44Xer3a3khpAWaVW/aSuHBDy1w/iOFXjuHubx8J4UiMnKgVjYJwZX48ok3WmbnV0cKDwJhTJ3oQMeZAT5ePMQEeq+BNpDLz7Q2VIiXBS/oaol04xaekLTr+UKW8RnmfcJWSnVwB6DE62UlWVLo6TxrDsr1lRi+LlhVH1bs2fiUxgS4daQ72DZ0ij0Q88UhB0U5skWTGpSKGk1akkpTXIl2glqIwaRddfBZ3mqtNvTF08xM6hQXFysZPXvpC+0uOvZWByecFk= ceph-0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:51:33.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsn6HBtunVth6g6h/031T+OePt3wxJ4wwtud1DeKFKBo9w9VhRd4keHgaYAsauPGrwkUHetay617yR8jpqp/yd/j+kMSed453l0/djiJ10wUi1L6q4dnggFZnU2TYQo7xt7Q12e6f8PlDT9Zyg0dHPAP6BgvQqDQboga4Oietu98QldfKZrgjVfcREBTmghcdSGrLBEgcUZEWUkvxYr7u1Me6tHMNyv+f6b7++jac2mDVeynyAd+MjusR44Xer3a3khpAWaVW/aSuHBDy1w/iOFXjuHubx8J4UiMnKgVjYJwZX48ok3WmbnV0cKDwJhTJ3oQMeZAT5ePMQEeq+BNpDLz7Q2VIiXBS/oaol04xaekLTr+UKW8RnmfcJWSnVwB6DE62UlWVLo6TxrDsr1lRi+LlhVH1bs2fiUxgS4daQ72DZ0ij0Q88UhB0U5skWTGpSKGk1akkpTXIl2glqIwaRddfBZ3mqtNvTF08xM6hQXFysZPXvpC+0uOvZWByecFk= ceph-0b28b95e-e01b-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T02:51:33.968 INFO:teuthology.orchestra.run.smithi129.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsn6HBtunVth6g6h/031T+OePt3wxJ4wwtud1DeKFKBo9w9VhRd4keHgaYAsauPGrwkUHetay617yR8jpqp/yd/j+kMSed453l0/djiJ10wUi1L6q4dnggFZnU2TYQo7xt7Q12e6f8PlDT9Zyg0dHPAP6BgvQqDQboga4Oietu98QldfKZrgjVfcREBTmghcdSGrLBEgcUZEWUkvxYr7u1Me6tHMNyv+f6b7++jac2mDVeynyAd+MjusR44Xer3a3khpAWaVW/aSuHBDy1w/iOFXjuHubx8J4UiMnKgVjYJwZX48ok3WmbnV0cKDwJhTJ3oQMeZAT5ePMQEeq+BNpDLz7Q2VIiXBS/oaol04xaekLTr+UKW8RnmfcJWSnVwB6DE62UlWVLo6TxrDsr1lRi+LlhVH1bs2fiUxgS4daQ72DZ0ij0Q88UhB0U5skWTGpSKGk1akkpTXIl2glqIwaRddfBZ3mqtNvTF08xM6hQXFysZPXvpC+0uOvZWByecFk= ceph-0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:51:33.983 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-12T02:51:34.076 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:33 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/3685942620' entity='client.admin' 2024-03-12T02:51:34.223 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:51:36.757 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:36 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:36.757 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:36 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:36.757 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:36 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:51:36.757 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:36 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:36.757 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:36 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:51:36.757 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:36 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:37.424 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-12T02:51:37.425 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-12T02:51:37.656 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:51:38.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:37 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/3616775718' entity='client.admin' 2024-03-12T02:51:38.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:37 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:51:38.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:37 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:40.131 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi115 2024-03-12T02:51:40.131 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:51:40.131 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.conf 2024-03-12T02:51:40.150 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:51:40.150 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:51:40.208 INFO:tasks.cephadm:Adding host smithi115 to orchestrator... 2024-03-12T02:51:40.208 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch host add smithi115 2024-03-12T02:51:40.448 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:51:40.669 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:40 smithi079 ceph-mon[25794]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:40.669 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:40.669 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:51:40.669 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:51:40.669 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:51:40.669 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:40 smithi079 ceph-mon[25794]: Updating smithi079:/etc/ceph/ceph.conf 2024-03-12T02:51:40.670 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:40.670 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:40.670 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:41.667 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:41 smithi079 ceph-mon[25794]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:51:41.667 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:41 smithi079 ceph-mon[25794]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:51:41.667 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:41 smithi079 ceph-mon[25794]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:51:42.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:42 smithi079 ceph-mon[25794]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi115", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:43.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:43 smithi079 ceph-mon[25794]: Deploying cephadm binary to smithi115 2024-03-12T02:51:45.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:45 smithi079 ceph-mon[25794]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:51:46.428 INFO:teuthology.orchestra.run.smithi079.stdout:Added host 'smithi115' with addr '172.21.15.115' 2024-03-12T02:51:46.964 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T02:51:47.200 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:51:47.572 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:47 smithi079 ceph-mon[25794]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:51:47.572 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:47 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:47.572 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:47 smithi079 ceph-mon[25794]: Added host smithi115 2024-03-12T02:51:47.572 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:47 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:51:47.572 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:47 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:48.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:48 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:49.025 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:49.026 INFO:teuthology.orchestra.run.smithi079.stdout:[{"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}] 2024-03-12T02:51:49.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:49 smithi079 ceph-mon[25794]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:51:49.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:49 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:49.644 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi129 2024-03-12T02:51:49.645 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:51:49.645 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.conf 2024-03-12T02:51:49.663 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:51:49.663 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:51:49.720 INFO:tasks.cephadm:Adding host smithi129 to orchestrator... 2024-03-12T02:51:49.720 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch host add smithi129 2024-03-12T02:51:49.967 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:51:50.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:50 smithi079 ceph-mon[25794]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T02:51:51.501 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:51 smithi079 ceph-mon[25794]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:51:53.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:51:53.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: Deploying cephadm binary to smithi129 2024-03-12T02:51:53.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:53.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:53.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:54.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:53 smithi079 ceph-mon[25794]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:51:55.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:54 smithi079 ceph-mon[25794]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:51:55.976 INFO:teuthology.orchestra.run.smithi079.stdout:Added host 'smithi129' with addr '172.21.15.129' 2024-03-12T02:51:56.540 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T02:51:56.772 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:51:57.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:56 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:57.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:56 smithi079 ceph-mon[25794]: Added host smithi129 2024-03-12T02:51:57.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:56 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:51:57.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:56 smithi079 ceph-mon[25794]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:51:57.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:56 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:58.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:57 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:58.641 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:51:58.642 INFO:teuthology.orchestra.run.smithi079.stdout:[{"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}, {"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}] 2024-03-12T02:51:59.215 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:58 smithi079 ceph-mon[25794]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:51:59.215 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:58 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:51:59.241 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-12T02:51:59.241 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-12T02:51:59.473 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:52:00.288 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:51:59 smithi079 ceph-mon[25794]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T02:52:01.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:00 smithi079 ceph-mon[25794]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:01.997 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-03-12T02:52:02.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:01 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1220296285' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T02:52:02.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:01 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:02.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:01 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:02.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:01 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:02.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:01 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:02.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:01 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:52:02.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:01 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:02.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:01 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:02.672 INFO:tasks.cephadm:Adding mon.a on smithi079 2024-03-12T02:52:02.672 INFO:tasks.cephadm:Adding mon.b on smithi115 2024-03-12T02:52:02.672 INFO:tasks.cephadm:Adding mon.c on smithi129 2024-03-12T02:52:02.673 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi079:172.21.15.79=a;smithi115:172.21.15.115=b;smithi129:172.21.15.129=c' 2024-03-12T02:52:02.895 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:03.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:02 smithi079 ceph-mon[25794]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-12T02:52:03.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:02 smithi079 ceph-mon[25794]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:03.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:02 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1220296285' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T02:52:03.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:02 smithi079 ceph-mon[25794]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:03.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:02 smithi079 ceph-mon[25794]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:52:03.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:02 smithi079 ceph-mon[25794]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:03.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:02 smithi079 ceph-mon[25794]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:52:03.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:02 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:03.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:02 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:03.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:02 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:04.506 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mon update... 2024-03-12T02:52:05.024 DEBUG:teuthology.orchestra.run.smithi115:mon.b> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.b.service 2024-03-12T02:52:05.027 DEBUG:teuthology.orchestra.run.smithi129:mon.c> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.c.service 2024-03-12T02:52:05.029 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T02:52:05.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T02:52:05.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:05 smithi079 ceph-mon[25794]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:05.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:05 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:05 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:05 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:05 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:05 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:05 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:05 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:06.768 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:06.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:06 smithi079 ceph-mon[25794]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi079:172.21.15.79=a;smithi115:172.21.15.115=b;smithi129:172.21.15.129=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:52:06.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:06 smithi079 ceph-mon[25794]: Saving service mon spec with placement smithi079:172.21.15.79=a;smithi115:172.21.15.115=b;smithi129:172.21.15.129=c;count:3 2024-03-12T02:52:06.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:06 smithi079 ceph-mon[25794]: Deploying daemon mon.c on smithi129 2024-03-12T02:52:07.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:07 smithi079 ceph-mon[25794]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:09.191 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:52:09.191 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":1,"fsid":"0b28b95e-e01b-11ee-95c8-87774f69a715","modified":"2024-03-12T02:50:28.956190Z","created":"2024-03-12T02:50:28.956190Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T02:52:09.193 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 1 2024-03-12T02:52:09.307 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:09 smithi129 systemd[1]: Started Ceph mon.c for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:52:10.947 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T02:52:10.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T02:52:11.193 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:52:14.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: mon.a calling monitor election 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: mon.c calling monitor election 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: monmap e2: 2 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],c=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T02:52:14.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: fsmap 2024-03-12T02:52:14.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:14.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: mgrmap e13: y(active, since 50s) 2024-03-12T02:52:14.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: overall HEALTH_OK 2024-03-12T02:52:14.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:14.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:14.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:14.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:14.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:14 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:14.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: mon.a calling monitor election 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: mon.c calling monitor election 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: monmap e2: 2 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],c=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: fsmap 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: mgrmap e13: y(active, since 50s) 2024-03-12T02:52:14.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: overall HEALTH_OK 2024-03-12T02:52:14.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:14.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:14.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:14.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:14.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:14 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:15.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:15 smithi079 ceph-mon[25794]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:15.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:15 smithi079 ceph-mon[25794]: Deploying daemon mon.b on smithi115 2024-03-12T02:52:15.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:15 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:15.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:15 smithi129 ceph-mon[30321]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:15.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:15 smithi129 ceph-mon[30321]: Deploying daemon mon.b on smithi115 2024-03-12T02:52:15.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:15 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:15.909 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:52:15.910 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":2,"fsid":"0b28b95e-e01b-11ee-95c8-87774f69a715","modified":"2024-03-12T02:52:09.289475Z","created":"2024-03-12T02:50:28.956190Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T02:52:15.912 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 2 2024-03-12T02:52:16.448 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:16 smithi129 ceph-mon[30321]: from='client.? 172.21.15.129:0/3724849127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T02:52:16.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:16 smithi079 ceph-mon[25794]: from='client.? 172.21.15.129:0/3724849127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T02:52:17.450 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T02:52:17.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T02:52:17.692 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:52:17.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:17 smithi079 ceph-mon[25794]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:17.813 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:17 smithi129 ceph-mon[30321]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.462 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(???) e0 preinit fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:52:18.462 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).mds e1 new map 2024-03-12T02:52:18.462 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-12T02:52:18.462 INFO:journalctl@ceph.mon.b.smithi115.stdout: e1 2024-03-12T02:52:18.462 INFO:journalctl@ceph.mon.b.smithi115.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T02:52:18.462 INFO:journalctl@ceph.mon.b.smithi115.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T02:52:18.462 INFO:journalctl@ceph.mon.b.smithi115.stdout: legacy client fscid: -1 2024-03-12T02:52:18.462 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-03-12T02:52:18.462 INFO:journalctl@ceph.mon.b.smithi115.stdout: No filesystems configured 2024-03-12T02:52:18.462 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.463 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Deploying cephadm binary to smithi129 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Added host smithi129 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.465 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1220296285' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T02:52:18.466 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.466 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.466 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.466 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.466 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:52:18.466 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:18.466 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:18.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-12T02:52:18.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:18.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1220296285' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T02:52:18.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:18.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:52:18.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi079:172.21.15.79=a;smithi115:172.21.15.115=b;smithi129:172.21.15.129=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:52:18.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Saving service mon spec with placement smithi079:172.21.15.79=a;smithi115:172.21.15.115=b;smithi129:172.21.15.129=c;count:3 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Deploying daemon mon.c on smithi129 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.a calling monitor election 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.c calling monitor election 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:18.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: monmap e2: 2 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],c=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: fsmap 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mgrmap e13: y(active, since 50s) 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: overall HEALTH_OK 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: Deploying daemon mon.b on smithi115 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: from='client.? 172.21.15.129:0/3724849127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T02:52:18.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:18.854 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 ceph-mon[29819]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-12T02:52:18.854 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 bash[29781]: 3e348d0eae3a378ffae3d0ce2825b5f8c28fc80a48bf54f9e4004f1b4fc2a86a 2024-03-12T02:52:18.854 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:18 smithi115 systemd[1]: Started Ceph mon.b for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:52:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:52:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: mon.a calling monitor election 2024-03-12T02:52:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: mon.c calling monitor election 2024-03-12T02:52:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: fsmap 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: mgrmap e13: y(active, since 59s) 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: mon.b (rank 2) addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] is down (out of quorum) 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:23.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:52:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: mon.a calling monitor election 2024-03-12T02:52:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: mon.c calling monitor election 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: fsmap 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: mgrmap e13: y(active, since 59s) 2024-03-12T02:52:23.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-12T02:52:23.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-12T02:52:23.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T02:52:23.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: mon.b (rank 2) addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] is down (out of quorum) 2024-03-12T02:52:23.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:23.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:24.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:24.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:24.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:24.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:24 smithi079 ceph-mon[25794]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:24.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:24.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:24.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:24 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:24.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:24 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:24.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:24 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:24.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:24 smithi129 ceph-mon[30321]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:24.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:24 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:24.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:24 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:25.374 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-12T02:52:25.374 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":3,"fsid":"0b28b95e-e01b-11ee-95c8-87774f69a715","modified":"2024-03-12T02:52:18.464410Z","created":"2024-03-12T02:50:28.956190Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T02:52:25.376 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 3 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: mon.a calling monitor election 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: mon.c calling monitor election 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:25.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: fsmap 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: mgrmap e13: y(active, since 59s) 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: mon.b (rank 2) addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] is down (out of quorum) 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:25.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:25.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:25.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:25.852 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:25.958 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-12T02:52:25.958 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-12T02:52:26.512 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: mon.b calling monitor election 2024-03-12T02:52:26.512 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: Updating smithi079:/etc/ceph/ceph.conf 2024-03-12T02:52:26.512 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-12T02:52:26.512 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-12T02:52:26.512 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:26.512 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:26.512 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:26.512 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: mon.b calling monitor election 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: mon.c calling monitor election 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: mon.a calling monitor election 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: fsmap 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: mgrmap e13: y(active, since 61s) 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: Cluster is now healthy 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: overall HEALTH_OK 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.513 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.514 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.514 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.514 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.514 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.514 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.514 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.514 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:26.514 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T02:52:26.514 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:26.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: mon.b calling monitor election 2024-03-12T02:52:26.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: Updating smithi079:/etc/ceph/ceph.conf 2024-03-12T02:52:26.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-12T02:52:26.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-12T02:52:26.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:26.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:26.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:26.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: mon.b calling monitor election 2024-03-12T02:52:26.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: mon.c calling monitor election 2024-03-12T02:52:26.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: mon.a calling monitor election 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: fsmap 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: mgrmap e13: y(active, since 61s) 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: Cluster is now healthy 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: overall HEALTH_OK 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:26.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T02:52:26.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:26.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: mon.b calling monitor election 2024-03-12T02:52:26.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: Updating smithi079:/etc/ceph/ceph.conf 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: mon.b calling monitor election 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: mon.c calling monitor election 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: mon.a calling monitor election 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: fsmap 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: mgrmap e13: y(active, since 61s) 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: Cluster is now healthy 2024-03-12T02:52:26.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: overall HEALTH_OK 2024-03-12T02:52:26.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.631 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:26.631 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:26.631 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T02:52:26.631 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:26.835 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:52:27.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:27 smithi079 ceph-mon[25794]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T02:52:27.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:27 smithi079 ceph-mon[25794]: Reconfiguring daemon mon.a on smithi079 2024-03-12T02:52:27.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:27 smithi079 ceph-mon[25794]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:27.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:27.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:27 smithi115 ceph-mon[29819]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T02:52:27.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:27 smithi115 ceph-mon[29819]: Reconfiguring daemon mon.a on smithi079 2024-03-12T02:52:27.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:27 smithi115 ceph-mon[29819]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:27.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:27 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:27.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:27 smithi129 ceph-mon[30321]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T02:52:27.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:27 smithi129 ceph-mon[30321]: Reconfiguring daemon mon.a on smithi079 2024-03-12T02:52:27.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:27 smithi129 ceph-mon[30321]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:27.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:27 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:52:28.735 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:28 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:28.735 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:28 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:28.735 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:28 smithi115 ceph-mon[29819]: Reconfiguring mon.b (monmap changed)... 2024-03-12T02:52:28.736 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:28 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:28.736 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:28 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T02:52:28.736 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:28 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:28.736 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:28 smithi115 ceph-mon[29819]: Reconfiguring daemon mon.b on smithi115 2024-03-12T02:52:28.736 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:28 smithi115 ceph-mon[29819]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:28.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:28 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:28.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:28 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:28.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:28 smithi079 ceph-mon[25794]: Reconfiguring mon.b (monmap changed)... 2024-03-12T02:52:28.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:28 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:28.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:28 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T02:52:28.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:28 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:28.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:28 smithi079 ceph-mon[25794]: Reconfiguring daemon mon.b on smithi115 2024-03-12T02:52:28.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:28 smithi079 ceph-mon[25794]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:28.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:28 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:28.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:28 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:28.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:28 smithi129 ceph-mon[30321]: Reconfiguring mon.b (monmap changed)... 2024-03-12T02:52:28.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:28 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:28.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:28 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T02:52:28.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:28 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:28.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:28 smithi129 ceph-mon[30321]: Reconfiguring daemon mon.b on smithi115 2024-03-12T02:52:28.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:28 smithi129 ceph-mon[30321]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:29.045 INFO:teuthology.orchestra.run.smithi079.stdout:# minimal ceph.conf for 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:52:29.045 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-03-12T02:52:29.046 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:52:29.046 INFO:teuthology.orchestra.run.smithi079.stdout: mon_host = [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2024-03-12T02:52:29.619 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:29 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/615991598' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:29.619 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:29 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:29.619 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:29 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:29.619 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:29 smithi079 ceph-mon[25794]: Reconfiguring mon.c (monmap changed)... 2024-03-12T02:52:29.619 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:29 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:29.619 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:29 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T02:52:29.619 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:29 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:29.619 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:29 smithi079 ceph-mon[25794]: Reconfiguring daemon mon.c on smithi129 2024-03-12T02:52:29.648 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-12T02:52:29.648 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:52:29.648 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T02:52:29.678 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:52:29.679 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:52:29.747 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:52:29.747 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T02:52:29.779 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:52:29.779 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:52:29.790 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:29 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/615991598' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:29.790 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:29 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:29.791 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:29 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:29.791 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:29 smithi129 ceph-mon[30321]: Reconfiguring mon.c (monmap changed)... 2024-03-12T02:52:29.791 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:29 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:29.791 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:29 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T02:52:29.791 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:29 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:29.791 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:29 smithi129 ceph-mon[30321]: Reconfiguring daemon mon.c on smithi129 2024-03-12T02:52:29.844 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:29 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/615991598' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:29.844 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:29 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:29.844 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:29 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:29.844 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:29 smithi115 ceph-mon[29819]: Reconfiguring mon.c (monmap changed)... 2024-03-12T02:52:29.844 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:29 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T02:52:29.844 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:29 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T02:52:29.844 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:29 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:29.845 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:29 smithi115 ceph-mon[29819]: Reconfiguring daemon mon.c on smithi129 2024-03-12T02:52:29.848 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:52:29.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T02:52:29.878 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:52:29.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:52:29.946 INFO:tasks.cephadm:Adding mgr.y on smithi079 2024-03-12T02:52:29.946 INFO:tasks.cephadm:Adding mgr.x on smithi115 2024-03-12T02:52:29.946 INFO:tasks.cephadm:Adding mgr.z on smithi129 2024-03-12T02:52:29.947 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi079=y;smithi115=x;smithi129=z' 2024-03-12T02:52:30.563 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:30 smithi129 ceph-mon[30321]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:30.626 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:52:30.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:30 smithi079 ceph-mon[25794]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:30.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:30 smithi115 ceph-mon[29819]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:31.770 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:31 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:31.770 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:31 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:31.770 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:31 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:31.770 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:31 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:31.770 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:31 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:31.770 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:31 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:32.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:31 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:32.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:31 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:32.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:31 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:32.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:31 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:32.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:31 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:32.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:31 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:31 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:31 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:31 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:31 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:31 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:31 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:32.334 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mgr update... 2024-03-12T02:52:32.861 DEBUG:teuthology.orchestra.run.smithi115:mgr.x> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.x.service 2024-03-12T02:52:32.863 DEBUG:teuthology.orchestra.run.smithi129:mgr.z> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.z.service 2024-03-12T02:52:32.865 INFO:tasks.cephadm:Deploying OSDs... 2024-03-12T02:52:32.865 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:52:32.865 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T02:52:32.884 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T02:52:32.884 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/nvme1n1 2024-03-12T02:52:32.942 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/nvme1n1 2024-03-12T02:52:32.942 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:32.942 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 761 Links: 1 Device type: 103,2 2024-03-12T02:52:32.942 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:32.942 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:32.942 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-12 02:51:34.463927723 +0000 2024-03-12T02:52:32.942 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-12 02:48:41.141926548 +0000 2024-03-12T02:52:32.942 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-12 02:48:41.141926548 +0000 2024-03-12T02:52:32.942 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-12 02:48:41.133926733 +0000 2024-03-12T02:52:32.943 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-12T02:52:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-12T02:52:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-12T02:52:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000375202 s, 1.4 MB/s 2024-03-12T02:52:33.013 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-12T02:52:33.071 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/nvme2n1 2024-03-12T02:52:33.127 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/nvme2n1 2024-03-12T02:52:33.127 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:33.128 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 773 Links: 1 Device type: 103,4 2024-03-12T02:52:33.128 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:33.128 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:33.128 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-12 02:51:34.474927470 +0000 2024-03-12T02:52:33.128 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-12 02:48:41.303922810 +0000 2024-03-12T02:52:33.128 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-12 02:48:41.303922810 +0000 2024-03-12T02:52:33.128 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-12 02:48:41.293923040 +0000 2024-03-12T02:52:33.128 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-12T02:52:33.195 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-12T02:52:33.195 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-12T02:52:33.195 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.0002483 s, 2.1 MB/s 2024-03-12T02:52:33.196 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-12T02:52:33.253 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/nvme3n1 2024-03-12T02:52:33.309 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/nvme3n1 2024-03-12T02:52:33.309 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:33.309 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 784 Links: 1 Device type: 103,6 2024-03-12T02:52:33.309 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:33.309 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:33.310 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-12 02:51:34.485927216 +0000 2024-03-12T02:52:33.310 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-12 02:48:41.414920248 +0000 2024-03-12T02:52:33.310 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-12 02:48:41.414920248 +0000 2024-03-12T02:52:33.310 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-12 02:48:41.407920409 +0000 2024-03-12T02:52:33.310 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-12T02:52:33.376 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-12T02:52:33.376 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-12T02:52:33.376 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000219161 s, 2.3 MB/s 2024-03-12T02:52:33.378 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-12T02:52:33.436 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/nvme4n1 2024-03-12T02:52:33.491 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/nvme4n1 2024-03-12T02:52:33.491 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:33.491 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 796 Links: 1 Device type: 103,8 2024-03-12T02:52:33.491 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:33.491 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:33.491 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-12 02:51:34.496926963 +0000 2024-03-12T02:52:33.491 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-12 02:48:41.569916670 +0000 2024-03-12T02:52:33.491 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-12 02:48:41.569916670 +0000 2024-03-12T02:52:33.492 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-12 02:48:41.558916924 +0000 2024-03-12T02:52:33.492 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-12T02:52:33.558 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-12T02:52:33.558 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-12T02:52:33.558 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000235956 s, 2.2 MB/s 2024-03-12T02:52:33.559 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-12T02:52:33.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi079=y;smithi115=x;smithi129=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: Saving service mgr spec with placement smithi079=y;smithi115=x;smithi129=z;count:3 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:33.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:33 smithi115 ceph-mon[29819]: Deploying daemon mgr.z on smithi129 2024-03-12T02:52:33.616 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:52:33.616 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T02:52:33.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:33.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi079=y;smithi115=x;smithi129=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:52:33.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: Saving service mgr spec with placement smithi079=y;smithi115=x;smithi129=z;count:3 2024-03-12T02:52:33.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:33.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:33.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:33.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:33.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:33.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T02:52:33.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T02:52:33.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T02:52:33.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:33.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:33 smithi129 ceph-mon[30321]: Deploying daemon mgr.z on smithi129 2024-03-12T02:52:33.635 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T02:52:33.635 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/nvme1n1 2024-03-12T02:52:33.693 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/nvme1n1 2024-03-12T02:52:33.693 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:33.693 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-12T02:52:33.693 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:33.693 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:33.694 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-12 02:51:50.948922246 +0000 2024-03-12T02:52:33.694 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-12 02:48:42.745469026 +0000 2024-03-12T02:52:33.694 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-12 02:48:42.745469026 +0000 2024-03-12T02:52:33.694 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-12 02:48:42.734469233 +0000 2024-03-12T02:52:33.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-12T02:52:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-12T02:52:33.763 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-12T02:52:33.764 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000296485 s, 1.7 MB/s 2024-03-12T02:52:33.765 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-12T02:52:33.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:33.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi079=y;smithi115=x;smithi129=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:52:33.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: Saving service mgr spec with placement smithi079=y;smithi115=x;smithi129=z;count:3 2024-03-12T02:52:33.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:33.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:33.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:33.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:33.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:33.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T02:52:33.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T02:52:33.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T02:52:33.811 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:33.812 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:33 smithi079 ceph-mon[25794]: Deploying daemon mgr.z on smithi129 2024-03-12T02:52:33.824 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/nvme2n1 2024-03-12T02:52:33.882 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/nvme2n1 2024-03-12T02:52:33.882 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:33.882 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-12T02:52:33.882 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:33.883 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:33.883 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-12 02:51:50.958922058 +0000 2024-03-12T02:52:33.883 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-12 02:48:42.897466161 +0000 2024-03-12T02:52:33.883 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-12 02:48:42.897466161 +0000 2024-03-12T02:52:33.883 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-12 02:48:42.890466293 +0000 2024-03-12T02:52:33.883 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-12T02:52:33.951 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-12T02:52:33.951 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-12T02:52:33.951 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000231719 s, 2.2 MB/s 2024-03-12T02:52:33.953 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-12T02:52:34.011 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/nvme3n1 2024-03-12T02:52:34.068 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/nvme3n1 2024-03-12T02:52:34.068 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:34.068 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-12T02:52:34.068 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:34.068 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:34.068 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-12 02:51:50.968921870 +0000 2024-03-12T02:52:34.068 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-12 02:48:43.047463333 +0000 2024-03-12T02:52:34.068 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-12 02:48:43.047463333 +0000 2024-03-12T02:52:34.068 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-12 02:48:43.036463540 +0000 2024-03-12T02:52:34.069 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-12T02:52:34.135 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-12T02:52:34.136 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-12T02:52:34.136 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000220801 s, 2.3 MB/s 2024-03-12T02:52:34.137 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-12T02:52:34.194 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/nvme4n1 2024-03-12T02:52:34.250 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/nvme4n1 2024-03-12T02:52:34.250 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:34.250 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-12T02:52:34.251 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:34.252 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:34.252 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-12 02:51:50.978921682 +0000 2024-03-12T02:52:34.252 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-12 02:48:43.197460506 +0000 2024-03-12T02:52:34.252 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-12 02:48:43.197460506 +0000 2024-03-12T02:52:34.252 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-12 02:48:43.185460732 +0000 2024-03-12T02:52:34.252 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-12T02:52:34.318 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-12T02:52:34.318 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-12T02:52:34.318 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000295665 s, 1.7 MB/s 2024-03-12T02:52:34.319 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-12T02:52:34.376 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:52:34.376 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T02:52:34.392 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T02:52:34.392 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/nvme1n1 2024-03-12T02:52:34.447 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/nvme1n1 2024-03-12T02:52:34.447 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:34.447 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-12T02:52:34.447 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:34.447 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:34.447 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-12 02:52:00.758471838 +0000 2024-03-12T02:52:34.447 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-12 02:48:44.412951630 +0000 2024-03-12T02:52:34.447 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-12 02:48:44.412951630 +0000 2024-03-12T02:52:34.448 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-12 02:48:44.400951905 +0000 2024-03-12T02:52:34.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-12T02:52:34.530 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-12T02:52:34.530 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-12T02:52:34.530 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000270822 s, 1.9 MB/s 2024-03-12T02:52:34.531 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-12T02:52:34.590 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/nvme2n1 2024-03-12T02:52:34.628 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:34 smithi129 systemd[1]: Starting Ceph mgr.z for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:52:34.643 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/nvme2n1 2024-03-12T02:52:34.643 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:34.643 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-12T02:52:34.643 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:34.643 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:34.644 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-12 02:52:00.769471589 +0000 2024-03-12T02:52:34.644 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-12 02:48:44.567948085 +0000 2024-03-12T02:52:34.644 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-12 02:48:44.567948085 +0000 2024-03-12T02:52:34.644 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-12 02:48:44.555948359 +0000 2024-03-12T02:52:34.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-12T02:52:34.710 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-12T02:52:34.710 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-12T02:52:34.710 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.00021823 s, 2.3 MB/s 2024-03-12T02:52:34.711 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-12T02:52:34.767 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/nvme3n1 2024-03-12T02:52:34.822 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/nvme3n1 2024-03-12T02:52:34.822 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:34.822 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-12T02:52:34.822 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:34.822 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:34.822 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-12 02:52:00.780471340 +0000 2024-03-12T02:52:34.822 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-12 02:48:44.668945774 +0000 2024-03-12T02:52:34.822 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-12 02:48:44.668945774 +0000 2024-03-12T02:52:34.822 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-12 02:48:44.662945911 +0000 2024-03-12T02:52:34.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-12T02:52:34.890 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-12T02:52:34.890 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-12T02:52:34.890 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000197283 s, 2.6 MB/s 2024-03-12T02:52:34.891 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-12T02:52:34.946 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/nvme4n1 2024-03-12T02:52:34.978 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:34 smithi129 podman[32269]: 2024-03-12 02:52:34.67065096 +0000 UTC m=+0.017729258 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T02:52:35.048 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:34 smithi129 podman[32269]: 2024-03-12 02:52:34.774279026 +0000 UTC m=+0.121357353 container create bbfacb7d3ec9929aaa0ebc08938981fd8c98cf157f1fb00ccf71d7c5760cd552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-12T02:52:35.049 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:34 smithi129 podman[32269]: 2024-03-12 02:52:34.976747293 +0000 UTC m=+0.323825578 container init bbfacb7d3ec9929aaa0ebc08938981fd8c98cf157f1fb00ccf71d7c5760cd552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T02:52:35.063 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/nvme4n1 2024-03-12T02:52:35.064 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T02:52:35.064 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-12T02:52:35.064 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T02:52:35.064 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T02:52:35.064 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-12 02:52:00.790471114 +0000 2024-03-12T02:52:35.064 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-12 02:48:44.815942411 +0000 2024-03-12T02:52:35.064 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-12 02:48:44.815942411 +0000 2024-03-12T02:52:35.064 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-12 02:48:44.810942525 +0000 2024-03-12T02:52:35.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-12T02:52:35.135 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-12T02:52:35.136 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-12T02:52:35.136 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000195595 s, 2.6 MB/s 2024-03-12T02:52:35.137 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-12T02:52:35.195 INFO:tasks.cephadm:Deploying osd.0 on smithi079 with /dev/nvme4n1... 2024-03-12T02:52:35.195 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-12T02:52:35.324 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:34 smithi129 podman[32269]: 2024-03-12 02:52:34.982323611 +0000 UTC m=+0.329401897 container start bbfacb7d3ec9929aaa0ebc08938981fd8c98cf157f1fb00ccf71d7c5760cd552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e) 2024-03-12T02:52:35.324 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-mgr[32397]: -- 172.21.15.129:0/4134452654 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a4669025a0 con 0x55a4668d6c00 2024-03-12T02:52:35.324 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:35 smithi129 bash[32269]: bbfacb7d3ec9929aaa0ebc08938981fd8c98cf157f1fb00ccf71d7c5760cd552 2024-03-12T02:52:35.324 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:35 smithi129 systemd[1]: Started Ceph mgr.z for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:52:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:52:35.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:35 smithi079 ceph-mon[25794]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:35.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:35 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:35.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:35 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:35.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:35 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:35.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:35 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T02:52:35.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:35 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T02:52:35.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:35 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T02:52:35.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:35 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:35.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:35 smithi115 ceph-mon[29819]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:35.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:35 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:35.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:35 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:35.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:35 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:35.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:35 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T02:52:35.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:35 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T02:52:35.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:35 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T02:52:35.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:35 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:35.628 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:35.567+0000 7fbb1ff37200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T02:52:35.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-mon[30321]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:35.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:35.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:35.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:35.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T02:52:35.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T02:52:35.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T02:52:35.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:36.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:35 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:35.660+0000 7fbb1ff37200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T02:52:36.374 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:36 smithi079 ceph-mon[25794]: Deploying daemon mgr.x on smithi115 2024-03-12T02:52:36.536 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:36 smithi129 ceph-mon[30321]: Deploying daemon mgr.x on smithi115 2024-03-12T02:52:36.536 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:36 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:36.302+0000 7fbb1ff37200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T02:52:36.536 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:36 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:36.393+0000 7fbb1ff37200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T02:52:36.536 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:36 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:36.533+0000 7fbb1ff37200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T02:52:36.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:36 smithi115 ceph-mon[29819]: Deploying daemon mgr.x on smithi115 2024-03-12T02:52:36.878 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:36 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:36.627+0000 7fbb1ff37200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T02:52:36.878 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:36 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:36.709+0000 7fbb1ff37200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T02:52:37.234 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/nvme4n1 2024-03-12T02:52:37.234 INFO:teuthology.orchestra.run.smithi079.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-12T02:52:37.234 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-12T02:52:37.234 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-03-12T02:52:37.234 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-03-12T02:52:37.234 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124006 s, 846 MB/s 2024-03-12T02:52:37.234 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-03-12T02:52:37.257 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:36 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:36.970+0000 7fbb1ff37200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T02:52:37.257 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:37 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:37.077+0000 7fbb1ff37200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T02:52:37.257 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:37 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:37.161+0000 7fbb1ff37200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T02:52:37.257 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:37 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:37.254+0000 7fbb1ff37200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T02:52:37.344 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:37 smithi115 systemd[1]: Starting Ceph mgr.x for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:52:37.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:37 smithi079 ceph-mon[25794]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:37.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:37 smithi115 ceph-mon[29819]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:37.600 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:37 smithi115 podman[31299]: 2024-03-12 02:52:37.341855207 +0000 UTC m=+0.113144222 container create 43d1f31507b4de219bcc9a4b88ededd8f5d8df86ba29f4c662bbbc0f271080d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-03-12T02:52:37.600 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:37 smithi115 podman[31299]: 2024-03-12 02:52:37.245877137 +0000 UTC m=+0.017166156 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T02:52:37.600 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:37 smithi115 podman[31299]: 2024-03-12 02:52:37.58596005 +0000 UTC m=+0.357249062 container init 43d1f31507b4de219bcc9a4b88ededd8f5d8df86ba29f4c662bbbc0f271080d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-03-12T02:52:37.600 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:37 smithi115 podman[31299]: 2024-03-12 02:52:37.591421036 +0000 UTC m=+0.362710041 container start 43d1f31507b4de219bcc9a4b88ededd8f5d8df86ba29f4c662bbbc0f271080d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-12T02:52:37.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:37 smithi129 ceph-mon[30321]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:37.628 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:37 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:37.430+0000 7fbb1ff37200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T02:52:37.858 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi079:/dev/nvme4n1 2024-03-12T02:52:38.094 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:52:38.099 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:37 smithi115 ceph-mgr[31336]: -- 172.21.15.115:0/3736883567 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557b289c85a0 con 0x557b2899c800 2024-03-12T02:52:38.099 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:37 smithi115 bash[31299]: 43d1f31507b4de219bcc9a4b88ededd8f5d8df86ba29f4c662bbbc0f271080d0 2024-03-12T02:52:38.099 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:37 smithi115 systemd[1]: Started Ceph mgr.x for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:52:38.599 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:38 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:38.146+0000 7fe3ce7d0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T02:52:38.599 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:38 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:38.243+0000 7fe3ce7d0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T02:52:38.741 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:38 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:38.425+0000 7fbb1ff37200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T02:52:38.741 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:38 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:38.568+0000 7fbb1ff37200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T02:52:38.804 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:38 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:38.804 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:38 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:38.804 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:38 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:38.804 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:38 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:38.804 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:38 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:38.804 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:38 smithi079 ceph-mon[25794]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:39.005 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:38 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:38.911+0000 7fbb1ff37200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T02:52:39.005 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:38 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:39.005 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:38 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:39.005 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:38 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:39.005 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:38 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:39.005 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:38 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:39.005 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:38 smithi129 ceph-mon[30321]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:39.032 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:38 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:38.936+0000 7fe3ce7d0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T02:52:39.032 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:39 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:39.030+0000 7fe3ce7d0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T02:52:39.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:38 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:39.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:38 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:39.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:38 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:39.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:38 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:39.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:38 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:39.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:38 smithi115 ceph-mon[29819]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:39.349 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:39 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:39.176+0000 7fe3ce7d0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T02:52:39.349 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:39 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:39.275+0000 7fe3ce7d0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T02:52:39.378 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:39 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:39.002+0000 7fbb1ff37200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T02:52:39.378 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:39 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:39.214+0000 7fbb1ff37200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:52:39.622 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:39 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:39.360+0000 7fe3ce7d0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T02:52:39.622 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:39 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:39.621+0000 7fe3ce7d0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T02:52:39.681 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:39 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:39.399+0000 7fbb1ff37200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T02:52:39.906 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:39 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:39.728+0000 7fe3ce7d0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T02:52:39.906 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:39 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:39.812+0000 7fe3ce7d0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T02:52:39.906 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:39 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:39.904+0000 7fe3ce7d0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T02:52:39.940 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:39 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:39.678+0000 7fbb1ff37200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:52:39.940 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:39 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:39.937+0000 7fbb1ff37200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T02:52:40.212 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:40.078+0000 7fe3ce7d0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T02:52:40.212 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:40.169+0000 7fbb1ff37200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T02:52:40.217 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.217 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.217 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.217 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.217 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:40.217 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:40.217 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.217 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.218 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T02:52:40.218 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T02:52:40.218 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T02:52:40.218 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:40.218 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: Reconfiguring daemon mgr.y on smithi079 2024-03-12T02:52:40.218 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T02:52:40.218 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T02:52:40.218 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:40 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:40.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:40.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:40.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T02:52:40.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T02:52:40.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T02:52:40.601 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:40.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: Reconfiguring daemon mgr.y on smithi079 2024-03-12T02:52:40.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T02:52:40.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T02:52:40.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:40 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:40.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: Reconfiguring daemon mgr.y on smithi079 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T02:52:40.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:41.221 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:40 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:40.952+0000 7fbb1ff37200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T02:52:41.349 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:41.052+0000 7fe3ce7d0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T02:52:41.350 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:41.182+0000 7fe3ce7d0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T02:52:41.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-mon[29819]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:52:41.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-mon[29819]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:41.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:41.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:41.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:41.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:41.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:41.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:41.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:41 smithi079 ceph-mon[25794]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:52:41.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:41 smithi079 ceph-mon[25794]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:41.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:41 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:41.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:41 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:41.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:41 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:41.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:41 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:41.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:41 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:41.407 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:41 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:41.525 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-mon[30321]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:52:41.526 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-mon[30321]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:41.526 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:41.526 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:41.526 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:41.526 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:41.526 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:41.526 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:41.796 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:41.511+0000 7fe3ce7d0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T02:52:41.796 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:41.599+0000 7fe3ce7d0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T02:52:41.796 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:41.793+0000 7fe3ce7d0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:52:41.833 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:41.523+0000 7fbb1ff37200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T02:52:41.833 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:41.619+0000 7fbb1ff37200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T02:52:41.833 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:52:41 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:52:41.830+0000 7fbb1ff37200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T02:52:42.099 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:41 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:41.974+0000 7fe3ce7d0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T02:52:42.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:42 smithi079 ceph-mon[25794]: Standby manager daemon z started 2024-03-12T02:52:42.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:42 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T02:52:42.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:42 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:52:42.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:42 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T02:52:42.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:42 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:52:42.515 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:42 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:42.256+0000 7fe3ce7d0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:52:42.515 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:42 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:42.512+0000 7fe3ce7d0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T02:52:42.515 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:42 smithi115 ceph-mon[29819]: Standby manager daemon z started 2024-03-12T02:52:42.515 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:42 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T02:52:42.516 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:42 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:52:42.516 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:42 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T02:52:42.516 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:42 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:52:42.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:42 smithi129 ceph-mon[30321]: Standby manager daemon z started 2024-03-12T02:52:42.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:42 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T02:52:42.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:42 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:52:42.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:42 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T02:52:42.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:42 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.129:0/3036803213' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:52:42.849 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:42 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:42.744+0000 7fe3ce7d0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T02:52:43.539 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:43 smithi115 ceph-mon[29819]: mgrmap e14: y(active, since 77s), standbys: z 2024-03-12T02:52:43.539 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:43 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T02:52:43.539 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:43 smithi115 ceph-mon[29819]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:43.539 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:43 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/707513824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fca3404-91d7-4899-8362-9b1c2a212438"}]: dispatch 2024-03-12T02:52:43.539 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:43 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/707513824' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fca3404-91d7-4899-8362-9b1c2a212438"}]': finished 2024-03-12T02:52:43.539 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:43 smithi115 ceph-mon[29819]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T02:52:43.539 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:43 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:52:43.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:43 smithi079 ceph-mon[25794]: mgrmap e14: y(active, since 77s), standbys: z 2024-03-12T02:52:43.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:43 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T02:52:43.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:43 smithi079 ceph-mon[25794]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:43.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:43 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/707513824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fca3404-91d7-4899-8362-9b1c2a212438"}]: dispatch 2024-03-12T02:52:43.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:43 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/707513824' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fca3404-91d7-4899-8362-9b1c2a212438"}]': finished 2024-03-12T02:52:43.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:43 smithi079 ceph-mon[25794]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T02:52:43.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:43 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:52:43.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:43 smithi129 ceph-mon[30321]: mgrmap e14: y(active, since 77s), standbys: z 2024-03-12T02:52:43.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:43 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T02:52:43.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:43 smithi129 ceph-mon[30321]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:43.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:43 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/707513824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fca3404-91d7-4899-8362-9b1c2a212438"}]: dispatch 2024-03-12T02:52:43.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:43 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/707513824' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fca3404-91d7-4899-8362-9b1c2a212438"}]': finished 2024-03-12T02:52:43.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:43 smithi129 ceph-mon[30321]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T02:52:43.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:43 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:52:43.849 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:43 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:43.536+0000 7fe3ce7d0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T02:52:44.349 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:44 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:44.096+0000 7fe3ce7d0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T02:52:44.350 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:44 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:44.191+0000 7fe3ce7d0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T02:52:44.849 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:52:44 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:52:44.403+0000 7fe3ce7d0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T02:52:45.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:45 smithi129 ceph-mon[30321]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:45.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:45 smithi129 ceph-mon[30321]: Standby manager daemon x started 2024-03-12T02:52:45.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:45 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T02:52:45.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:45 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:52:45.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:45 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T02:52:45.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:45 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:52:45.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:45 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/3357151952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T02:52:45.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:45 smithi079 ceph-mon[25794]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:45.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:45 smithi079 ceph-mon[25794]: Standby manager daemon x started 2024-03-12T02:52:45.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:45 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T02:52:45.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:45 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:52:45.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:45 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T02:52:45.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:45 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:52:45.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:45 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/3357151952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T02:52:45.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:45 smithi115 ceph-mon[29819]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:45.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:45 smithi115 ceph-mon[29819]: Standby manager daemon x started 2024-03-12T02:52:45.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:45 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T02:52:45.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:45 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:52:45.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:45 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T02:52:45.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:45 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.115:0/819359654' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:52:45.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:45 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/3357151952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T02:52:46.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:46 smithi115 ceph-mon[29819]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-12T02:52:46.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:46 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T02:52:46.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:46 smithi129 ceph-mon[30321]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-12T02:52:46.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:46 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T02:52:46.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:46 smithi079 ceph-mon[25794]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-12T02:52:46.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:46 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T02:52:47.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:47 smithi079 ceph-mon[25794]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:47.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:47 smithi129 ceph-mon[30321]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:47.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:47 smithi115 ceph-mon[29819]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:49.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:49 smithi079 ceph-mon[25794]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:49.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:49 smithi129 ceph-mon[30321]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:49.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:49 smithi115 ceph-mon[29819]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:50.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:50 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T02:52:50.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:50 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:50.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:50 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T02:52:50.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:50 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:50.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:50 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T02:52:50.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:50 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:51.488 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:51 smithi079 ceph-mon[25794]: Deploying daemon osd.0 on smithi079 2024-03-12T02:52:51.488 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:51 smithi079 ceph-mon[25794]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:51.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:51 smithi129 ceph-mon[30321]: Deploying daemon osd.0 on smithi079 2024-03-12T02:52:51.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:51 smithi129 ceph-mon[30321]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:51.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:51 smithi115 ceph-mon[29819]: Deploying daemon osd.0 on smithi079 2024-03-12T02:52:51.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:51 smithi115 ceph-mon[29819]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:53.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:53 smithi079 ceph-mon[25794]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:53.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:53 smithi115 ceph-mon[29819]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:53.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:53 smithi129 ceph-mon[30321]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:55.620 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:55 smithi079 ceph-mon[25794]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:55.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:55 smithi115 ceph-mon[29819]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:55.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:55 smithi129 ceph-mon[30321]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:57.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:56 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:57.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:56 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:57.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:56 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:57.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:56 smithi129 ceph-mon[30321]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:57.212 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:56 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:57.212 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:56 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:57.212 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:56 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:57.213 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:56 smithi079 ceph-mon[25794]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:57.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:56 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:57.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:56 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:57.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:56 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:57.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:56 smithi115 ceph-mon[29819]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:59.255 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 0 on host 'smithi079' 2024-03-12T02:52:59.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:59 smithi079 ceph-mon[25794]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:59.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:59.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:59.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:59 smithi079 ceph-mon[25794]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T02:52:59.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:52:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:59.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:59 smithi129 ceph-mon[30321]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:59.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:59.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:59.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:59 smithi129 ceph-mon[30321]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T02:52:59.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:52:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:59.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:59 smithi115 ceph-mon[29819]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:52:59.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:52:59.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:52:59.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:59 smithi115 ceph-mon[29819]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T02:52:59.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:52:59.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:52:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:52:59.906 DEBUG:teuthology.orchestra.run.smithi079:osd.0> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.0.service 2024-03-12T02:52:59.908 INFO:tasks.cephadm:Deploying osd.1 on smithi115 with /dev/nvme4n1... 2024-03-12T02:52:59.909 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-12T02:53:00.152 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T02:53:00.968 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:00 smithi115 ceph-mon[29819]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T02:53:00.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:00 smithi115 ceph-mon[29819]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T02:53:00.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:00 smithi115 ceph-mon[29819]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-12T02:53:00.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:00 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:00.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:00 smithi115 ceph-mon[29819]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:53:01.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:00 smithi079 ceph-mon[25794]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T02:53:01.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:00 smithi079 ceph-mon[25794]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T02:53:01.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:00 smithi079 ceph-mon[25794]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-12T02:53:01.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:00 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:01.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:00 smithi079 ceph-mon[25794]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:53:01.060 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:53:00 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0[39977]: 2024-03-12T02:53:00.761+0000 7f6d5a481700 -1 osd.0 0 waiting for initial osdmap 2024-03-12T02:53:01.060 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:53:00 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0[39977]: 2024-03-12T02:53:00.765+0000 7f6d56274700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T02:53:01.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:00 smithi129 ceph-mon[30321]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T02:53:01.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:00 smithi129 ceph-mon[30321]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T02:53:01.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:00 smithi129 ceph-mon[30321]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-03-12T02:53:01.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:00 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:01.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:00 smithi129 ceph-mon[30321]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:53:02.051 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/nvme4n1 2024-03-12T02:53:02.051 INFO:teuthology.orchestra.run.smithi115.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-12T02:53:02.051 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-12T02:53:02.051 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2024-03-12T02:53:02.051 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2024-03-12T02:53:02.051 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123609 s, 848 MB/s 2024-03-12T02:53:02.051 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2024-03-12T02:53:02.051 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:01 smithi115 ceph-mon[29819]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-12T02:53:02.052 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:01 smithi115 ceph-mon[29819]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T02:53:02.052 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:01 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:02.052 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:01 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:02.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:01 smithi079 ceph-mon[25794]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-12T02:53:02.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:01 smithi079 ceph-mon[25794]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T02:53:02.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:01 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:02.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:01 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:02.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:01 smithi129 ceph-mon[30321]: from='osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-03-12T02:53:02.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:01 smithi129 ceph-mon[30321]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T02:53:02.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:01 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:02.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:01 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:02.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi115:/dev/nvme4n1 2024-03-12T02:53:02.864 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:02 smithi079 ceph-mon[25794]: purged_snaps scrub starts 2024-03-12T02:53:02.865 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:02 smithi079 ceph-mon[25794]: purged_snaps scrub ok 2024-03-12T02:53:02.865 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:02 smithi079 ceph-mon[25794]: osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361] boot 2024-03-12T02:53:02.865 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:02 smithi079 ceph-mon[25794]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T02:53:02.865 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:02 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:02.865 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:02 smithi079 ceph-mon[25794]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:53:02.927 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T02:53:03.049 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:02 smithi115 ceph-mon[29819]: purged_snaps scrub starts 2024-03-12T02:53:03.049 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:02 smithi115 ceph-mon[29819]: purged_snaps scrub ok 2024-03-12T02:53:03.049 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:02 smithi115 ceph-mon[29819]: osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361] boot 2024-03-12T02:53:03.049 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:02 smithi115 ceph-mon[29819]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T02:53:03.049 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:02 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:03.049 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:02 smithi115 ceph-mon[29819]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:53:03.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:02 smithi129 ceph-mon[30321]: purged_snaps scrub starts 2024-03-12T02:53:03.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:02 smithi129 ceph-mon[30321]: purged_snaps scrub ok 2024-03-12T02:53:03.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:02 smithi129 ceph-mon[30321]: osd.0 [v2:172.21.15.79:6802/388305361,v1:172.21.15.79:6803/388305361] boot 2024-03-12T02:53:03.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:02 smithi129 ceph-mon[30321]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T02:53:03.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:02 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:53:03.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:02 smithi129 ceph-mon[30321]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T02:53:04.030 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:03 smithi129 ceph-mon[30321]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T02:53:04.030 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:03 smithi129 ceph-mon[30321]: Detected new or changed devices on smithi079 2024-03-12T02:53:04.030 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.030 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.030 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:53:04.030 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:03 smithi129 ceph-mon[30321]: Adjusting osd_memory_target on smithi079 to 17088M 2024-03-12T02:53:04.030 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.030 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:04.030 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:04.031 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.039 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:03 smithi115 ceph-mon[29819]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T02:53:04.040 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:03 smithi115 ceph-mon[29819]: Detected new or changed devices on smithi079 2024-03-12T02:53:04.040 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.040 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.040 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:53:04.040 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:03 smithi115 ceph-mon[29819]: Adjusting osd_memory_target on smithi079 to 17088M 2024-03-12T02:53:04.040 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.040 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:04.040 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:04.040 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:03 smithi079 ceph-mon[25794]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T02:53:04.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:03 smithi079 ceph-mon[25794]: Detected new or changed devices on smithi079 2024-03-12T02:53:04.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:53:04.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:03 smithi079 ceph-mon[25794]: Adjusting osd_memory_target on smithi079 to 17088M 2024-03-12T02:53:04.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:04.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:04.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:04.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:05.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:04 smithi079 ceph-mon[25794]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:05.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:04 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T02:53:05.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:04 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T02:53:05.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:04 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:05.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:04 smithi115 ceph-mon[29819]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:05.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:04 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T02:53:05.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:04 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T02:53:05.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:04 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:05.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:04 smithi129 ceph-mon[30321]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:05.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:04 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T02:53:05.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:04 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T02:53:05.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:04 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:06.039 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:05 smithi115 ceph-mon[29819]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:53:06.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:05 smithi079 ceph-mon[25794]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:53:06.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:05 smithi129 ceph-mon[30321]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:53:06.866 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:06 smithi115 ceph-mon[29819]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:07.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:06 smithi079 ceph-mon[25794]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:07.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:06 smithi129 ceph-mon[30321]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:08.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:07 smithi079 ceph-mon[25794]: from='client.? 172.21.15.115:0/3881203643' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79286417-821f-4092-a811-f3eaea1907bf"}]: dispatch 2024-03-12T02:53:08.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:07 smithi079 ceph-mon[25794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79286417-821f-4092-a811-f3eaea1907bf"}]: dispatch 2024-03-12T02:53:08.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:07 smithi079 ceph-mon[25794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79286417-821f-4092-a811-f3eaea1907bf"}]': finished 2024-03-12T02:53:08.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:07 smithi079 ceph-mon[25794]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T02:53:08.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:07 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:08.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:07 smithi115 ceph-mon[29819]: from='client.? 172.21.15.115:0/3881203643' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79286417-821f-4092-a811-f3eaea1907bf"}]: dispatch 2024-03-12T02:53:08.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:07 smithi115 ceph-mon[29819]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79286417-821f-4092-a811-f3eaea1907bf"}]: dispatch 2024-03-12T02:53:08.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:07 smithi115 ceph-mon[29819]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79286417-821f-4092-a811-f3eaea1907bf"}]': finished 2024-03-12T02:53:08.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:07 smithi115 ceph-mon[29819]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T02:53:08.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:07 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:08.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:07 smithi129 ceph-mon[30321]: from='client.? 172.21.15.115:0/3881203643' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79286417-821f-4092-a811-f3eaea1907bf"}]: dispatch 2024-03-12T02:53:08.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:07 smithi129 ceph-mon[30321]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79286417-821f-4092-a811-f3eaea1907bf"}]: dispatch 2024-03-12T02:53:08.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:07 smithi129 ceph-mon[30321]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79286417-821f-4092-a811-f3eaea1907bf"}]': finished 2024-03-12T02:53:08.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:07 smithi129 ceph-mon[30321]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T02:53:08.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:07 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:09.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:08 smithi079 ceph-mon[25794]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:09.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:08 smithi079 ceph-mon[25794]: from='client.? 172.21.15.115:0/1405067030' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T02:53:09.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:08 smithi115 ceph-mon[29819]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:09.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:08 smithi115 ceph-mon[29819]: from='client.? 172.21.15.115:0/1405067030' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T02:53:09.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:08 smithi129 ceph-mon[30321]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:09.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:08 smithi129 ceph-mon[30321]: from='client.? 172.21.15.115:0/1405067030' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T02:53:11.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:11 smithi129 ceph-mon[30321]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:11.684 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:11 smithi115 ceph-mon[29819]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:11.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:11 smithi079 ceph-mon[25794]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:13.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:13 smithi115 ceph-mon[29819]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:13.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:13 smithi129 ceph-mon[30321]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:13.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:13 smithi079 ceph-mon[25794]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:15.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:15 smithi115 ceph-mon[29819]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:15.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:15 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T02:53:15.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:15 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:15.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:15 smithi129 ceph-mon[30321]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:15.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:15 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T02:53:15.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:15 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:15.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:15 smithi079 ceph-mon[25794]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:15.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:15 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T02:53:15.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:15 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:16.518 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:16 smithi115 ceph-mon[29819]: Deploying daemon osd.1 on smithi115 2024-03-12T02:53:16.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:16 smithi129 ceph-mon[30321]: Deploying daemon osd.1 on smithi115 2024-03-12T02:53:16.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:16 smithi079 ceph-mon[25794]: Deploying daemon osd.1 on smithi115 2024-03-12T02:53:17.451 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:17 smithi115 ceph-mon[29819]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:17.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:17 smithi129 ceph-mon[30321]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:17.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:17 smithi079 ceph-mon[25794]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:19.593 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:19 smithi115 ceph-mon[29819]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:19.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:19 smithi129 ceph-mon[30321]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:19.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:19 smithi079 ceph-mon[25794]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:21.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:21 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:21.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:21 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:21.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:21 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:21.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:21 smithi115 ceph-mon[29819]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:21.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:21 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:21.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:21 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:21.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:21 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:21.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:21 smithi079 ceph-mon[25794]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:21.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:21 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:21.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:21 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:21.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:21 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:21.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:21 smithi129 ceph-mon[30321]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:23.453 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:23 smithi115 ceph-mon[29819]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:23.453 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:23 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:23.453 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:23 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:23.453 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:23 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:23.453 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:23 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:23.453 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:23 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:23.453 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:23 smithi115 ceph-mon[29819]: from='osd.1 [v2:172.21.15.115:6800/2621660585,v1:172.21.15.115:6801/2621660585]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T02:53:23.453 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:23 smithi115 ceph-mon[29819]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T02:53:23.518 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 1 on host 'smithi115' 2024-03-12T02:53:23.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:23 smithi129 ceph-mon[30321]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:23.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:23.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:23.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:23.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:23.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:23.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:23 smithi129 ceph-mon[30321]: from='osd.1 [v2:172.21.15.115:6800/2621660585,v1:172.21.15.115:6801/2621660585]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T02:53:23.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:23 smithi129 ceph-mon[30321]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T02:53:23.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:23 smithi079 ceph-mon[25794]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:23.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:23.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:23.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:23 smithi079 ceph-mon[25794]: from='osd.1 [v2:172.21.15.115:6800/2621660585,v1:172.21.15.115:6801/2621660585]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T02:53:23.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:23 smithi079 ceph-mon[25794]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T02:53:24.128 DEBUG:teuthology.orchestra.run.smithi115:osd.1> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.1.service 2024-03-12T02:53:24.130 INFO:tasks.cephadm:Deploying osd.2 on smithi129 with /dev/nvme4n1... 2024-03-12T02:53:24.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-12T02:53:24.376 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:53:24.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:24.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:24.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:24.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:24 smithi079 ceph-mon[25794]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T02:53:24.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:24 smithi079 ceph-mon[25794]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T02:53:24.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:24.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:24 smithi079 ceph-mon[25794]: from='osd.1 [v2:172.21.15.115:6800/2621660585,v1:172.21.15.115:6801/2621660585]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-12T02:53:24.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:24 smithi079 ceph-mon[25794]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-12T02:53:24.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:24 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:24.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:24 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:24.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:24 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:24.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:24 smithi115 ceph-mon[29819]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T02:53:24.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:24 smithi115 ceph-mon[29819]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T02:53:24.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:24 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:24.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:24 smithi115 ceph-mon[29819]: from='osd.1 [v2:172.21.15.115:6800/2621660585,v1:172.21.15.115:6801/2621660585]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-12T02:53:24.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:24 smithi115 ceph-mon[29819]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-12T02:53:24.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:24 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:24.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:24 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:24.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:24 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:24.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:24 smithi129 ceph-mon[30321]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T02:53:24.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:24 smithi129 ceph-mon[30321]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T02:53:24.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:24 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:24.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:24 smithi129 ceph-mon[30321]: from='osd.1 [v2:172.21.15.115:6800/2621660585,v1:172.21.15.115:6801/2621660585]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-12T02:53:24.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:24 smithi129 ceph-mon[30321]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-12T02:53:25.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:25 smithi115 ceph-mon[29819]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:25.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:25 smithi115 ceph-mon[29819]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-12T02:53:25.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:25 smithi115 ceph-mon[29819]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T02:53:25.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:25.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:25 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:25.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:25 smithi115 ceph-mon[29819]: from='osd.1 ' entity='osd.1' 2024-03-12T02:53:25.600 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:53:25 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1[34422]: 2024-03-12T02:53:25.374+0000 7f19e608f700 -1 osd.1 0 waiting for initial osdmap 2024-03-12T02:53:25.601 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:53:25 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1[34422]: 2024-03-12T02:53:25.377+0000 7f19dd679700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T02:53:25.807 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:25 smithi129 ceph-mon[30321]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:25.807 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:25 smithi129 ceph-mon[30321]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-12T02:53:25.807 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:25 smithi129 ceph-mon[30321]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T02:53:25.807 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:25 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:25.808 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:25 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:25.808 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:25 smithi129 ceph-mon[30321]: from='osd.1 ' entity='osd.1' 2024-03-12T02:53:25.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:25 smithi079 ceph-mon[25794]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:25.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:25 smithi079 ceph-mon[25794]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-12T02:53:25.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:25 smithi079 ceph-mon[25794]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T02:53:25.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:25 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:25.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:25 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:25.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:25 smithi079 ceph-mon[25794]: from='osd.1 ' entity='osd.1' 2024-03-12T02:53:26.234 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/nvme4n1 2024-03-12T02:53:26.235 INFO:teuthology.orchestra.run.smithi129.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-12T02:53:26.235 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-12T02:53:26.235 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-12T02:53:26.235 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-12T02:53:26.235 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124491 s, 842 MB/s 2024-03-12T02:53:26.235 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-12T02:53:26.760 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:26 smithi129 ceph-mon[30321]: purged_snaps scrub starts 2024-03-12T02:53:26.760 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:26 smithi129 ceph-mon[30321]: purged_snaps scrub ok 2024-03-12T02:53:26.760 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:26.760 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:26 smithi129 ceph-mon[30321]: osd.1 [v2:172.21.15.115:6800/2621660585,v1:172.21.15.115:6801/2621660585] boot 2024-03-12T02:53:26.761 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:26 smithi129 ceph-mon[30321]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T02:53:26.761 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:26 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:26.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi129:/dev/nvme4n1 2024-03-12T02:53:26.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:26 smithi079 ceph-mon[25794]: purged_snaps scrub starts 2024-03-12T02:53:26.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:26 smithi079 ceph-mon[25794]: purged_snaps scrub ok 2024-03-12T02:53:26.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:26.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:26 smithi079 ceph-mon[25794]: osd.1 [v2:172.21.15.115:6800/2621660585,v1:172.21.15.115:6801/2621660585] boot 2024-03-12T02:53:26.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:26 smithi079 ceph-mon[25794]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T02:53:26.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:26 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:26.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:26 smithi115 ceph-mon[29819]: purged_snaps scrub starts 2024-03-12T02:53:26.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:26 smithi115 ceph-mon[29819]: purged_snaps scrub ok 2024-03-12T02:53:26.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:26.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:26 smithi115 ceph-mon[29819]: osd.1 [v2:172.21.15.115:6800/2621660585,v1:172.21.15.115:6801/2621660585] boot 2024-03-12T02:53:26.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:26 smithi115 ceph-mon[29819]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T02:53:26.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:26 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:53:27.020 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:53:27.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:27 smithi079 ceph-mon[25794]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:27.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:53:27.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:27.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:27.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:27 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.842 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:27 smithi129 ceph-mon[30321]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:27.842 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:27 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.842 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:27 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.842 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:27 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:53:27.842 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:27 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.842 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:27 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:27.842 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:27 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:27.842 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:27 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:27 smithi115 ceph-mon[29819]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T02:53:27.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:27 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:27 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:27 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:53:27.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:27 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:27.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:27 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:27.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:27 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:27.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:27 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:28.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:28 smithi079 ceph-mon[25794]: Detected new or changed devices on smithi115 2024-03-12T02:53:28.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:28 smithi079 ceph-mon[25794]: Adjusting osd_memory_target on smithi115 to 17088M 2024-03-12T02:53:28.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:28 smithi079 ceph-mon[25794]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T02:53:28.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:28 smithi115 ceph-mon[29819]: Detected new or changed devices on smithi115 2024-03-12T02:53:28.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:28 smithi115 ceph-mon[29819]: Adjusting osd_memory_target on smithi115 to 17088M 2024-03-12T02:53:28.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:28 smithi115 ceph-mon[29819]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T02:53:28.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:28 smithi129 ceph-mon[30321]: Detected new or changed devices on smithi115 2024-03-12T02:53:28.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:28 smithi129 ceph-mon[30321]: Adjusting osd_memory_target on smithi115 to 17088M 2024-03-12T02:53:28.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:28 smithi129 ceph-mon[30321]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T02:53:29.688 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:29 smithi129 ceph-mon[30321]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:29.689 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:29 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T02:53:29.689 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:29 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T02:53:29.689 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:29 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:29.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:29 smithi079 ceph-mon[25794]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:29.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:29 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T02:53:29.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:29 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T02:53:29.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:29 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:29.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:29 smithi115 ceph-mon[29819]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:29.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:29 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T02:53:29.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:29 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T02:53:29.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:29 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:30.570 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:30 smithi129 ceph-mon[30321]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:53:30.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:30 smithi079 ceph-mon[25794]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:53:30.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:30 smithi115 ceph-mon[29819]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:53:31.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:31 smithi079 ceph-mon[25794]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:31.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:31 smithi079 ceph-mon[25794]: from='client.? 172.21.15.129:0/1660613541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "354aa3b0-d113-4978-b0db-614c813242d8"}]: dispatch 2024-03-12T02:53:31.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:31 smithi079 ceph-mon[25794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "354aa3b0-d113-4978-b0db-614c813242d8"}]: dispatch 2024-03-12T02:53:31.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:31 smithi079 ceph-mon[25794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "354aa3b0-d113-4978-b0db-614c813242d8"}]': finished 2024-03-12T02:53:31.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:31 smithi079 ceph-mon[25794]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T02:53:31.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:31 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:31.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:31 smithi115 ceph-mon[29819]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:31.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:31 smithi115 ceph-mon[29819]: from='client.? 172.21.15.129:0/1660613541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "354aa3b0-d113-4978-b0db-614c813242d8"}]: dispatch 2024-03-12T02:53:31.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:31 smithi115 ceph-mon[29819]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "354aa3b0-d113-4978-b0db-614c813242d8"}]: dispatch 2024-03-12T02:53:31.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:31 smithi115 ceph-mon[29819]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "354aa3b0-d113-4978-b0db-614c813242d8"}]': finished 2024-03-12T02:53:31.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:31 smithi115 ceph-mon[29819]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T02:53:31.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:31 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:31.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:31 smithi129 ceph-mon[30321]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:31.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:31 smithi129 ceph-mon[30321]: from='client.? 172.21.15.129:0/1660613541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "354aa3b0-d113-4978-b0db-614c813242d8"}]: dispatch 2024-03-12T02:53:31.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:31 smithi129 ceph-mon[30321]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "354aa3b0-d113-4978-b0db-614c813242d8"}]: dispatch 2024-03-12T02:53:31.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:31 smithi129 ceph-mon[30321]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "354aa3b0-d113-4978-b0db-614c813242d8"}]': finished 2024-03-12T02:53:31.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:31 smithi129 ceph-mon[30321]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T02:53:31.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:31 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:32.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:32 smithi079 ceph-mon[25794]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:32.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:32 smithi115 ceph-mon[29819]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:32.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:32 smithi129 ceph-mon[30321]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:33.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:33 smithi079 ceph-mon[25794]: from='client.? 172.21.15.129:0/3260954099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T02:53:33.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:33 smithi115 ceph-mon[29819]: from='client.? 172.21.15.129:0/3260954099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T02:53:33.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:33 smithi129 ceph-mon[30321]: from='client.? 172.21.15.129:0/3260954099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T02:53:34.778 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:34 smithi129 ceph-mon[30321]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:34.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:34 smithi079 ceph-mon[25794]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:34.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:34 smithi115 ceph-mon[29819]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:37.539 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:37 smithi129 ceph-mon[30321]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:37.796 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:37 smithi115 ceph-mon[29819]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:37.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:37 smithi079 ceph-mon[25794]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:39.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:39 smithi129 ceph-mon[30321]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:39.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:39 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T02:53:39.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:39 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:39.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:39 smithi129 ceph-mon[30321]: Deploying daemon osd.2 on smithi129 2024-03-12T02:53:39.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:39 smithi079 ceph-mon[25794]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:39.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:39 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T02:53:39.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:39 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:39.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:39 smithi079 ceph-mon[25794]: Deploying daemon osd.2 on smithi129 2024-03-12T02:53:39.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:39 smithi115 ceph-mon[29819]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:39.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:39 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T02:53:39.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:39 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:39.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:39 smithi115 ceph-mon[29819]: Deploying daemon osd.2 on smithi129 2024-03-12T02:53:41.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:41 smithi129 ceph-mon[30321]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:41.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:41 smithi079 ceph-mon[25794]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:41.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:41 smithi115 ceph-mon[29819]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:43.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:43 smithi129 ceph-mon[30321]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:43.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:43 smithi079 ceph-mon[25794]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:43.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:43 smithi115 ceph-mon[29819]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:45.206 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:44 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:45.206 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:44 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:45.206 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:44 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:45.206 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:44 smithi129 ceph-mon[30321]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:45.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:44 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:45.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:44 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:45.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:44 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:45.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:44 smithi079 ceph-mon[25794]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:45.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:44 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:45.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:44 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:45.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:44 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:45.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:44 smithi115 ceph-mon[29819]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:47.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:47 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:47.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:47 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:47.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:47 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:47.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:47 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:47.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:47 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:47.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:47 smithi129 ceph-mon[30321]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:47.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:47 smithi129 ceph-mon[30321]: from='osd.2 [v2:172.21.15.129:6800/2648140063,v1:172.21.15.129:6801/2648140063]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T02:53:47.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:47 smithi129 ceph-mon[30321]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T02:53:47.467 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 2 on host 'smithi129' 2024-03-12T02:53:47.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:47 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:47.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:47 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:47.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:47 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:47.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:47 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:47.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:47 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:47.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:47 smithi079 ceph-mon[25794]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:47.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:47 smithi079 ceph-mon[25794]: from='osd.2 [v2:172.21.15.129:6800/2648140063,v1:172.21.15.129:6801/2648140063]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T02:53:47.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:47 smithi079 ceph-mon[25794]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T02:53:47.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:47 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:47.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:47 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:47.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:47 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:47.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:47 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:47.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:47 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:47.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:47 smithi115 ceph-mon[29819]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:47.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:47 smithi115 ceph-mon[29819]: from='osd.2 [v2:172.21.15.129:6800/2648140063,v1:172.21.15.129:6801/2648140063]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T02:53:47.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:47 smithi115 ceph-mon[29819]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T02:53:48.012 DEBUG:teuthology.orchestra.run.smithi129:osd.2> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.2.service 2024-03-12T02:53:48.014 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-12T02:53:48.015 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-12T02:53:48.260 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:53:48.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:48 smithi079 ceph-mon[25794]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T02:53:48.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:48 smithi079 ceph-mon[25794]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T02:53:48.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:48 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:48.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:48 smithi079 ceph-mon[25794]: from='osd.2 [v2:172.21.15.129:6800/2648140063,v1:172.21.15.129:6801/2648140063]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-12T02:53:48.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:48 smithi079 ceph-mon[25794]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-12T02:53:48.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:48 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:48.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:48 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:48.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:48 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:48.319 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:48 smithi129 ceph-mon[30321]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T02:53:48.319 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:48 smithi129 ceph-mon[30321]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T02:53:48.320 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:48 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:48.320 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:48 smithi129 ceph-mon[30321]: from='osd.2 [v2:172.21.15.129:6800/2648140063,v1:172.21.15.129:6801/2648140063]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-12T02:53:48.320 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:48 smithi129 ceph-mon[30321]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-12T02:53:48.320 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:48 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:48.320 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:48 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:48.320 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:48 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:48.320 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:53:48 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2[35611]: 2024-03-12T02:53:48.218+0000 7f71ad08b700 -1 osd.2 0 waiting for initial osdmap 2024-03-12T02:53:48.320 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:53:48 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2[35611]: 2024-03-12T02:53:48.221+0000 7f71a8e7e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T02:53:48.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:48 smithi115 ceph-mon[29819]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T02:53:48.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:48 smithi115 ceph-mon[29819]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T02:53:48.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:48 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:48.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:48 smithi115 ceph-mon[29819]: from='osd.2 [v2:172.21.15.129:6800/2648140063,v1:172.21.15.129:6801/2648140063]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-12T02:53:48.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:48 smithi115 ceph-mon[29819]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-12T02:53:48.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:48 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:48.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:48 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:48.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:48 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:49.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:49 smithi079 ceph-mon[25794]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-12T02:53:49.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:49 smithi079 ceph-mon[25794]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T02:53:49.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:49 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:49.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:49 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:49.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:49 smithi079 ceph-mon[25794]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:49.530 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:49 smithi129 ceph-mon[30321]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-12T02:53:49.530 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:49 smithi129 ceph-mon[30321]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T02:53:49.531 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:49 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:49.531 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:49 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:49.531 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:49 smithi129 ceph-mon[30321]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:49.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:49 smithi115 ceph-mon[29819]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-12T02:53:49.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:49 smithi115 ceph-mon[29819]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T02:53:49.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:49 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:49.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:49 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:49.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:49 smithi115 ceph-mon[29819]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T02:53:50.238 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:53:50.509 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:50 smithi129 ceph-mon[30321]: purged_snaps scrub starts 2024-03-12T02:53:50.510 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:50 smithi129 ceph-mon[30321]: purged_snaps scrub ok 2024-03-12T02:53:50.510 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:50 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:50.510 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:50 smithi129 ceph-mon[30321]: osd.2 [v2:172.21.15.129:6800/2648140063,v1:172.21.15.129:6801/2648140063] boot 2024-03-12T02:53:50.510 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:50 smithi129 ceph-mon[30321]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T02:53:50.510 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:50 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:50.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:50 smithi079 ceph-mon[25794]: purged_snaps scrub starts 2024-03-12T02:53:50.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:50 smithi079 ceph-mon[25794]: purged_snaps scrub ok 2024-03-12T02:53:50.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:50 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:50.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:50 smithi079 ceph-mon[25794]: osd.2 [v2:172.21.15.129:6800/2648140063,v1:172.21.15.129:6801/2648140063] boot 2024-03-12T02:53:50.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:50 smithi079 ceph-mon[25794]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T02:53:50.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:50 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:50.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:50 smithi115 ceph-mon[29819]: purged_snaps scrub starts 2024-03-12T02:53:50.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:50 smithi115 ceph-mon[29819]: purged_snaps scrub ok 2024-03-12T02:53:50.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:50 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:50.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:50 smithi115 ceph-mon[29819]: osd.2 [v2:172.21.15.129:6800/2648140063,v1:172.21.15.129:6801/2648140063] boot 2024-03-12T02:53:50.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:50 smithi115 ceph-mon[29819]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T02:53:50.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:50 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:53:50.889 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":19,"num_osds":3,"num_up_osds":3,"osd_up_since":1710212029,"num_in_osds":3,"osd_in_since":1710212011,"num_remapped_pgs":0} 2024-03-12T02:53:50.890 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T02:53:51.120 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:53:51.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T02:53:51.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1456481610' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T02:53:51.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: pgmap v82: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:51.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T02:53:51.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:53:51.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:51.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:51.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:51 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T02:53:51.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1456481610' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T02:53:51.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: pgmap v82: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:51.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T02:53:51.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:53:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:51 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T02:53:51.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1456481610' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T02:53:51.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: pgmap v82: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:51.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T02:53:51.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:53:51.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:51.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:51.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:51.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:51 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:52.499 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:52 smithi079 ceph-mon[25794]: Detected new or changed devices on smithi129 2024-03-12T02:53:52.500 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:52 smithi079 ceph-mon[25794]: Adjusting osd_memory_target on smithi129 to 17088M 2024-03-12T02:53:52.500 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T02:53:52.500 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:52 smithi079 ceph-mon[25794]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T02:53:52.500 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:52 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T02:53:52.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:52 smithi115 ceph-mon[29819]: Detected new or changed devices on smithi129 2024-03-12T02:53:52.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:52 smithi115 ceph-mon[29819]: Adjusting osd_memory_target on smithi129 to 17088M 2024-03-12T02:53:52.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:52 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T02:53:52.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:52 smithi115 ceph-mon[29819]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T02:53:52.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:52 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T02:53:52.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:52 smithi129 ceph-mon[30321]: Detected new or changed devices on smithi129 2024-03-12T02:53:52.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:52 smithi129 ceph-mon[30321]: Adjusting osd_memory_target on smithi129 to 17088M 2024-03-12T02:53:52.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:52 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T02:53:52.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:52 smithi129 ceph-mon[30321]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T02:53:52.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:52 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T02:53:52.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:53:52 smithi079 sudo[45403]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-12T02:53:52.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:53:52 smithi079 sudo[45403]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:52.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:53:52 smithi079 sudo[45403]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:52.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:53:52 smithi079 sudo[45403]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:52.930 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:53:52.930 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":21,"fsid":"0b28b95e-e01b-11ee-95c8-87774f69a715","created":"2024-03-12T02:50:32.098344+0000","modified":"2024-03-12T02:53:52.234230+0000","last_up_change":"2024-03-12T02:53:49.215820+0000","last_in_change":"2024-03-12T02:53:31.439447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T02:53:50.612844+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fca3404-91d7-4899-8362-9b1c2a212438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6803","nonce":388305361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6805","nonce":388305361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6809","nonce":388305361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6807","nonce":388305361}]},"public_addr":"172.21.15.79:6803/388305361","cluster_addr":"172.21.15.79:6805/388305361","heartbeat_back_addr":"172.21.15.79:6809/388305361","heartbeat_front_addr":"172.21.15.79:6807/388305361","state":["exists","up"]},{"osd":1,"uuid":"79286417-821f-4092-a811-f3eaea1907bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6801","nonce":2621660585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6803","nonce":2621660585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6807","nonce":2621660585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6805","nonce":2621660585}]},"public_addr":"172.21.15.115:6801/2621660585","cluster_addr":"172.21.15.115:6803/2621660585","heartbeat_back_addr":"172.21.15.115:6807/2621660585","heartbeat_front_addr":"172.21.15.115:6805/2621660585","state":["exists","up"]},{"osd":2,"uuid":"354aa3b0-d113-4978-b0db-614c813242d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6801","nonce":2648140063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6803","nonce":2648140063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6807","nonce":2648140063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6805","nonce":2648140063}]},"public_addr":"172.21.15.129:6801/2648140063","cluster_addr":"172.21.15.129:6803/2648140063","heartbeat_back_addr":"172.21.15.129:6807/2648140063","heartbeat_front_addr":"172.21.15.129:6805/2648140063","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T02:52:59.965800+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T02:53:24.223303+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T02:53:48.054301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1172559344":"2024-03-13T02:51:24.282856+0000","172.21.15.79:0/3747240":"2024-03-13T02:51:24.282856+0000","172.21.15.79:6800/2289827629":"2024-03-13T02:51:24.282856+0000","172.21.15.79:6801/2289827629":"2024-03-13T02:51:24.282856+0000","172.21.15.79:0/2417320756":"2024-03-13T02:50:58.528795+0000","172.21.15.79:0/1664099364":"2024-03-13T02:51:24.282856+0000","172.21.15.79:6800/1227173847":"2024-03-13T02:50:58.528795+0000","172.21.15.79:0/2524931850":"2024-03-13T02:50:58.528795+0000","172.21.15.79:6801/1227173847":"2024-03-13T02:50:58.528795+0000","172.21.15.79:0/3251309801":"2024-03-13T02:50:58.528795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T02:53:53.246 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:53:52 smithi079 sudo[45433]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-12T02:53:53.247 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:53:52 smithi079 sudo[45433]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:53.247 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:53:52 smithi079 sudo[45433]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:53.247 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:53:52 smithi079 sudo[45433]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:53.370 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:53:53 smithi115 sudo[39843]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-12T02:53:53.370 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:53:53 smithi115 sudo[39843]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:53.370 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:53:53 smithi115 sudo[39843]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:53.370 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:53:53 smithi115 sudo[39843]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:53.370 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:53 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T02:53:53.370 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:53 smithi115 ceph-mon[29819]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T02:53:53.370 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:53 smithi115 ceph-mon[29819]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:53.371 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:53 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:53.371 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:53 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/698235436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T02:53:53.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:53 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T02:53:53.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:53 smithi079 ceph-mon[25794]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T02:53:53.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:53 smithi079 ceph-mon[25794]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:53.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:53 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:53.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:53 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/698235436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T02:53:53.621 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T02:53:50.612844+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T02:53:53.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-12T02:53:53.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:53 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T02:53:53.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:53 smithi129 ceph-mon[30321]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T02:53:53.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:53 smithi129 ceph-mon[30321]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:53.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:53 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:53.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:53 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/698235436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T02:53:53.849 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:53:53 smithi115 sudo[39846]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-12T02:53:53.849 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:53:53 smithi115 sudo[39846]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:53.849 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:53:53 smithi115 sudo[39846]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:53.850 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:53:53 smithi115 sudo[39846]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:53.857 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:53:53.890 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:53:53 smithi129 sudo[41136]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-12T02:53:53.890 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:53:53 smithi129 sudo[41136]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:53.890 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:53:53 smithi129 sudo[41136]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:53.890 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:53:53 smithi129 sudo[41136]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:54.296 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 sudo[45552]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T02:53:54.296 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 sudo[45552]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:54.296 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 sudo[45552]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:54.296 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 sudo[45552]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:54.378 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:53:53 smithi129 sudo[41139]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-12T02:53:54.378 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:53:53 smithi129 sudo[41139]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:54.378 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:53:53 smithi129 sudo[41139]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:54.378 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:53:53 smithi129 sudo[41139]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:54.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 sudo[45573]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T02:53:54.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 sudo[45573]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:54.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 sudo[45573]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:54.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 sudo[45573]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:54.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 ceph-mon[25794]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T02:53:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 ceph-mon[25794]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-12T02:53:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:54 smithi079 ceph-mon[25794]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T02:53:54.703 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 ceph-mon[30321]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T02:53:54.703 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:54.703 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 ceph-mon[30321]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-12T02:53:54.703 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:54.703 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 ceph-mon[30321]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T02:53:54.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:54 smithi115 ceph-mon[29819]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T02:53:54.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:54 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:54.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:54 smithi115 ceph-mon[29819]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-12T02:53:54.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:54 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:54.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:54 smithi115 ceph-mon[29819]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T02:53:54.955 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 sudo[41142]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T02:53:54.956 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 sudo[41142]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:54.956 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 sudo[41142]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:54.956 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 sudo[41142]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:55.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 sudo[41145]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T02:53:55.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 sudo[41145]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:55.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 sudo[41145]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:55.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:54 smithi129 sudo[41145]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:55.480 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 sudo[39849]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T02:53:55.480 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 sudo[39849]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:55.480 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 sudo[39849]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:55.480 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 sudo[39849]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:55.649 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:55.650 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:53:55.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:53:55.651 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:55 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 sudo[39852]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 sudo[39852]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 sudo[39852]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 sudo[39852]: pam_unix(sudo:session): session closed for user root 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:55.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T02:53:55.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:53:55.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:53:55.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:53:55.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T02:53:55.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:55.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:53:55.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:53:55.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:55 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:53:55.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:55.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T02:53:55.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:53:55.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:53:55.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:53:55.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:55.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T02:53:55.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:53:55.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:53:55.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:53:55.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T02:53:55.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:55.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:53:55.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:53:55.879 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:55 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:53:56.191 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi079 2024-03-12T02:53:56.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi079=foo.a' 2024-03-12T02:53:56.440 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:53:56.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:56 smithi079 ceph-mon[25794]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T02:53:56.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:56 smithi079 ceph-mon[25794]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T02:53:56.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:56 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:56.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:56 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2324691425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T02:53:56.813 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:56 smithi129 ceph-mon[30321]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T02:53:56.813 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:56 smithi129 ceph-mon[30321]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T02:53:56.813 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:56 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:56.813 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:56 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2324691425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T02:53:56.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:56 smithi115 ceph-mon[29819]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T02:53:56.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:56 smithi115 ceph-mon[29819]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T02:53:56.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:56 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:56.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:56 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2324691425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T02:53:57.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:57 smithi129 ceph-mon[30321]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:57.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:57 smithi079 ceph-mon[25794]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:57.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:57 smithi115 ceph-mon[29819]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:58.194 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled rgw.foo.a update... 2024-03-12T02:53:58.733 DEBUG:teuthology.orchestra.run.smithi079:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-12T02:53:58.737 INFO:tasks.cephadm:Adding prometheus.a on smithi115 2024-03-12T02:53:58.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi115=a' 2024-03-12T02:53:58.969 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:53:59.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi079=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:53:59.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-03-12T02:53:59.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:59.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:59.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:59.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:59.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:59.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:59.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T02:53:59.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T02:53:59.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:59.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:53:59 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:59.518 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi079=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:53:59.518 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-03-12T02:53:59.518 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:59.518 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:59.518 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:59.518 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:59.518 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:59.519 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:59.519 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T02:53:59.519 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T02:53:59.519 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:59.519 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:53:59 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi079=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:53:59.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:53:59 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:54:00.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:00 smithi079 ceph-mon[25794]: Deploying daemon rgw.foo.a on smithi079 2024-03-12T02:54:00.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:00 smithi115 ceph-mon[29819]: Deploying daemon rgw.foo.a on smithi079 2024-03-12T02:54:00.600 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled prometheus update... 2024-03-12T02:54:00.600 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:00 smithi129 ceph-mon[30321]: Deploying daemon rgw.foo.a on smithi079 2024-03-12T02:54:01.060 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:54:00 smithi079 systemd[1]: Starting Ceph rgw.foo.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:54:01.174 DEBUG:teuthology.orchestra.run.smithi115:prometheus.a> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-12T02:54:01.177 INFO:tasks.cephadm:Adding node-exporter.a on smithi079 2024-03-12T02:54:01.177 INFO:tasks.cephadm:Adding node-exporter.b on smithi115 2024-03-12T02:54:01.177 INFO:tasks.cephadm:Adding node-exporter.c on smithi129 2024-03-12T02:54:01.177 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '3;smithi079=a;smithi115=b;smithi129=c' 2024-03-12T02:54:01.392 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:01 smithi079 ceph-mon[25794]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:01.392 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:01 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:01.393 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:54:01 smithi079 podman[45992]: 2024-03-12 02:54:00.97650911 +0000 UTC m=+0.017358958 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T02:54:01.393 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:54:01 smithi079 podman[45992]: 2024-03-12 02:54:01.08876595 +0000 UTC m=+0.129615786 container create 7d29761e6c9d3fe6a7d3a987af555fe2292d3873bfd6d7067f609bb00f2b485a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T02:54:01.393 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:54:01 smithi079 podman[45992]: 2024-03-12 02:54:01.299461274 +0000 UTC m=+0.340311122 container init 7d29761e6c9d3fe6a7d3a987af555fe2292d3873bfd6d7067f609bb00f2b485a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-12T02:54:01.393 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:54:01 smithi079 podman[45992]: 2024-03-12 02:54:01.30472953 +0000 UTC m=+0.345579374 container start 7d29761e6c9d3fe6a7d3a987af555fe2292d3873bfd6d7067f609bb00f2b485a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-rgw-foo-a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T02:54:01.393 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:54:01 smithi079 bash[45992]: 7d29761e6c9d3fe6a7d3a987af555fe2292d3873bfd6d7067f609bb00f2b485a 2024-03-12T02:54:01.413 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:54:01.528 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:01 smithi129 ceph-mon[30321]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:01.528 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:01 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:01.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:01 smithi115 ceph-mon[29819]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:01.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:01 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:01.809 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:54:01 smithi079 systemd[1]: Started Ceph rgw.foo.a for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:54:02.498 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:02 smithi129 ceph-mon[30321]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:02.498 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:02 smithi129 ceph-mon[30321]: Saving service prometheus spec with placement smithi115=a;count:1 2024-03-12T02:54:02.498 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:02 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.499 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:02 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.499 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:02 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.499 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:02 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.499 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:02 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.499 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:02 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:54:02.545 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:02 smithi079 ceph-mon[25794]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:02.545 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:02 smithi079 ceph-mon[25794]: Saving service prometheus spec with placement smithi115=a;count:1 2024-03-12T02:54:02.545 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:02 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.545 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:02 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.545 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:02 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.545 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:02 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.545 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:02 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:02 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:54:02.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:02 smithi115 ceph-mon[29819]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:02.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:02 smithi115 ceph-mon[29819]: Saving service prometheus spec with placement smithi115=a;count:1 2024-03-12T02:54:02.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:02 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:02 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:02 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:02 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:02 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:02.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:02 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:54:03.026 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled node-exporter update... 2024-03-12T02:54:03.364 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:03 smithi129 ceph-mon[30321]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-03-12T02:54:03.365 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:03 smithi129 ceph-mon[30321]: osdmap e23: 3 total, 3 up, 3 in 2024-03-12T02:54:03.365 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:03 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1839683242' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T02:54:03.365 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:03 smithi129 ceph-mon[30321]: pgmap v92: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:03.365 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:03.365 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:03.365 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:03.365 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:54:03.365 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:03 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:54:03.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:03 smithi079 ceph-mon[25794]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-03-12T02:54:03.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:03 smithi079 ceph-mon[25794]: osdmap e23: 3 total, 3 up, 3 in 2024-03-12T02:54:03.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:03 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1839683242' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T02:54:03.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:03 smithi079 ceph-mon[25794]: pgmap v92: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:03.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:03.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:03.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:03.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:54:03.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:03 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:54:03.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:03 smithi115 ceph-mon[29819]: Saving service rgw.foo.a spec with placement smithi079=foo.a;count:1 2024-03-12T02:54:03.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:03 smithi115 ceph-mon[29819]: osdmap e23: 3 total, 3 up, 3 in 2024-03-12T02:54:03.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:03 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1839683242' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T02:54:03.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:03 smithi115 ceph-mon[29819]: pgmap v92: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:03.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:03.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:03.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:03.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:54:03.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:03 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:54:03.648 DEBUG:teuthology.orchestra.run.smithi079:node-exporter.a> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-12T02:54:03.651 DEBUG:teuthology.orchestra.run.smithi115:node-exporter.b> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-12T02:54:03.653 DEBUG:teuthology.orchestra.run.smithi129:node-exporter.c> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@node-exporter.c.service 2024-03-12T02:54:03.655 INFO:tasks.cephadm:Adding alertmanager.a on smithi079 2024-03-12T02:54:03.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi079=a' 2024-03-12T02:54:03.923 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:54:04.318 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:04 smithi129 ceph-mon[30321]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi079=a;smithi115=b;smithi129=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:04.318 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:04 smithi129 ceph-mon[30321]: Saving service node-exporter spec with placement smithi079=a;smithi115=b;smithi129=c;count:3 2024-03-12T02:54:04.318 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:04 smithi129 ceph-mon[30321]: Checking dashboard <-> RGW credentials 2024-03-12T02:54:04.318 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:04 smithi129 ceph-mon[30321]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T02:54:04.318 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:04 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1839683242' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T02:54:04.319 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:04 smithi129 ceph-mon[30321]: osdmap e24: 3 total, 3 up, 3 in 2024-03-12T02:54:04.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:04 smithi079 ceph-mon[25794]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi079=a;smithi115=b;smithi129=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:04.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:04 smithi079 ceph-mon[25794]: Saving service node-exporter spec with placement smithi079=a;smithi115=b;smithi129=c;count:3 2024-03-12T02:54:04.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:04 smithi079 ceph-mon[25794]: Checking dashboard <-> RGW credentials 2024-03-12T02:54:04.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:04 smithi079 ceph-mon[25794]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T02:54:04.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:04 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1839683242' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T02:54:04.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:04 smithi079 ceph-mon[25794]: osdmap e24: 3 total, 3 up, 3 in 2024-03-12T02:54:04.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:04 smithi115 ceph-mon[29819]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi079=a;smithi115=b;smithi129=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:04.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:04 smithi115 ceph-mon[29819]: Saving service node-exporter spec with placement smithi079=a;smithi115=b;smithi129=c;count:3 2024-03-12T02:54:04.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:04 smithi115 ceph-mon[29819]: Checking dashboard <-> RGW credentials 2024-03-12T02:54:04.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:04 smithi115 ceph-mon[29819]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T02:54:04.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:04 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1839683242' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T02:54:04.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:04 smithi115 ceph-mon[29819]: osdmap e24: 3 total, 3 up, 3 in 2024-03-12T02:54:05.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:05 smithi129 ceph-mon[30321]: osdmap e25: 3 total, 3 up, 3 in 2024-03-12T02:54:05.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:05 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T02:54:05.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:05 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T02:54:05.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:05 smithi129 ceph-mon[30321]: pgmap v95: 65 pgs: 8 creating+peering, 1 active+clean, 56 unknown; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:05.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:05 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:05.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:05 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T02:54:05.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:05 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T02:54:05.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:05 smithi129 ceph-mon[30321]: osdmap e26: 3 total, 3 up, 3 in 2024-03-12T02:54:05.755 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled alertmanager update... 2024-03-12T02:54:05.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:05 smithi079 ceph-mon[25794]: osdmap e25: 3 total, 3 up, 3 in 2024-03-12T02:54:05.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:05 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T02:54:05.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:05 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T02:54:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:05 smithi079 ceph-mon[25794]: pgmap v95: 65 pgs: 8 creating+peering, 1 active+clean, 56 unknown; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:05 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:05 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T02:54:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:05 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T02:54:05.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:05 smithi079 ceph-mon[25794]: osdmap e26: 3 total, 3 up, 3 in 2024-03-12T02:54:05.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:05 smithi115 ceph-mon[29819]: osdmap e25: 3 total, 3 up, 3 in 2024-03-12T02:54:05.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:05 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T02:54:05.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:05 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T02:54:05.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:05 smithi115 ceph-mon[29819]: pgmap v95: 65 pgs: 8 creating+peering, 1 active+clean, 56 unknown; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:05.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:05 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:05.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:05 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T02:54:05.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:05 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T02:54:05.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:05 smithi115 ceph-mon[29819]: osdmap e26: 3 total, 3 up, 3 in 2024-03-12T02:54:06.289 DEBUG:teuthology.orchestra.run.smithi079:alertmanager.a> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-12T02:54:06.291 INFO:tasks.cephadm:Adding grafana.a on smithi115 2024-03-12T02:54:06.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi115=a' 2024-03-12T02:54:06.537 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:54:06.932 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:06 smithi129 ceph-mon[30321]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi079=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:06.932 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:06 smithi129 ceph-mon[30321]: Saving service alertmanager spec with placement smithi079=a;count:1 2024-03-12T02:54:06.932 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:06 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:06.932 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:06 smithi129 ceph-mon[30321]: osdmap e27: 3 total, 3 up, 3 in 2024-03-12T02:54:06.933 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:06 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T02:54:06.933 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:06 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T02:54:06.933 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:06 smithi129 ceph-mon[30321]: pgmap v98: 97 pgs: 17 creating+peering, 15 active+clean, 65 unknown; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T02:54:07.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:06 smithi079 ceph-mon[25794]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi079=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:07.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:06 smithi079 ceph-mon[25794]: Saving service alertmanager spec with placement smithi079=a;count:1 2024-03-12T02:54:07.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:06 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:07.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:06 smithi079 ceph-mon[25794]: osdmap e27: 3 total, 3 up, 3 in 2024-03-12T02:54:07.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:06 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T02:54:07.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:06 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T02:54:07.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:06 smithi079 ceph-mon[25794]: pgmap v98: 97 pgs: 17 creating+peering, 15 active+clean, 65 unknown; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T02:54:07.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:06 smithi115 ceph-mon[29819]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi079=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:07.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:06 smithi115 ceph-mon[29819]: Saving service alertmanager spec with placement smithi079=a;count:1 2024-03-12T02:54:07.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:06 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:07.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:06 smithi115 ceph-mon[29819]: osdmap e27: 3 total, 3 up, 3 in 2024-03-12T02:54:07.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:06 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T02:54:07.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:06 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T02:54:07.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:06 smithi115 ceph-mon[29819]: pgmap v98: 97 pgs: 17 creating+peering, 15 active+clean, 65 unknown; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-12T02:54:08.447 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled grafana update... 2024-03-12T02:54:08.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:08 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T02:54:08.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:08 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T02:54:08.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:08 smithi079 ceph-mon[25794]: osdmap e28: 3 total, 3 up, 3 in 2024-03-12T02:54:08.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:08 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T02:54:08.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:08 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T02:54:08.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:08 smithi115 ceph-mon[29819]: osdmap e28: 3 total, 3 up, 3 in 2024-03-12T02:54:08.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:08 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T02:54:08.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:08 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T02:54:08.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:08 smithi129 ceph-mon[30321]: osdmap e28: 3 total, 3 up, 3 in 2024-03-12T02:54:08.969 DEBUG:teuthology.orchestra.run.smithi115:grafana.a> sudo journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@grafana.a.service 2024-03-12T02:54:08.972 INFO:tasks.cephadm:Setting up client nodes... 2024-03-12T02:54:08.973 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T02:54:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:09.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:09 smithi079 ceph-mon[25794]: osdmap e29: 3 total, 3 up, 3 in 2024-03-12T02:54:09.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:09 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T02:54:09.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:09 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T02:54:09.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:09 smithi079 ceph-mon[25794]: pgmap v101: 129 pgs: 18 creating+peering, 67 active+clean, 44 unknown; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-12T02:54:09.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:09 smithi079 ceph-mon[25794]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:09.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:09 smithi079 ceph-mon[25794]: Saving service grafana spec with placement smithi115=a;count:1 2024-03-12T02:54:09.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:09 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:09.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:09 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T02:54:09.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:09 smithi115 ceph-mon[29819]: osdmap e29: 3 total, 3 up, 3 in 2024-03-12T02:54:09.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:09 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T02:54:09.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:09 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T02:54:09.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:09 smithi115 ceph-mon[29819]: pgmap v101: 129 pgs: 18 creating+peering, 67 active+clean, 44 unknown; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-12T02:54:09.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:09 smithi115 ceph-mon[29819]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:09.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:09 smithi115 ceph-mon[29819]: Saving service grafana spec with placement smithi115=a;count:1 2024-03-12T02:54:09.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:09 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:09.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:09 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T02:54:09.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:09 smithi129 ceph-mon[30321]: osdmap e29: 3 total, 3 up, 3 in 2024-03-12T02:54:09.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:09 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T02:54:09.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:09 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T02:54:09.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:09 smithi129 ceph-mon[30321]: pgmap v101: 129 pgs: 18 creating+peering, 67 active+clean, 44 unknown; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-12T02:54:09.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:09 smithi129 ceph-mon[30321]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:09.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:09 smithi129 ceph-mon[30321]: Saving service grafana spec with placement smithi115=a;count:1 2024-03-12T02:54:09.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:09 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:09.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:09 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T02:54:10.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:10 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T02:54:10.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:10 smithi079 ceph-mon[25794]: osdmap e30: 3 total, 3 up, 3 in 2024-03-12T02:54:10.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:10 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T02:54:10.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:10 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T02:54:10.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:10 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T02:54:10.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:10 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T02:54:10.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:10 smithi079 ceph-mon[25794]: osdmap e31: 3 total, 3 up, 3 in 2024-03-12T02:54:10.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:10 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T02:54:10.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:10 smithi115 ceph-mon[29819]: osdmap e30: 3 total, 3 up, 3 in 2024-03-12T02:54:10.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:10 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T02:54:10.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:10 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T02:54:10.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:10 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T02:54:10.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:10 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T02:54:10.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:10 smithi115 ceph-mon[29819]: osdmap e31: 3 total, 3 up, 3 in 2024-03-12T02:54:10.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:10 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T02:54:10.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:10 smithi129 ceph-mon[30321]: osdmap e30: 3 total, 3 up, 3 in 2024-03-12T02:54:10.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:10 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T02:54:10.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:10 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T02:54:10.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:10 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2172731666' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T02:54:10.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:10 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1324017989' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T02:54:10.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:10 smithi129 ceph-mon[30321]: osdmap e31: 3 total, 3 up, 3 in 2024-03-12T02:54:11.199 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-03-12T02:54:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDTw+9l/n2uCxAAqHZYgFZ/yrzPWxBEYHAbyg== 2024-03-12T02:54:11.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:11 smithi079 ceph-mon[25794]: pgmap v104: 129 pgs: 9 creating+peering, 97 active+clean, 23 unknown; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T02:54:11.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:11 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:11.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:11 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:11.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:11 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:11.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:11 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/3560754543' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:11.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:11 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/3560754543' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T02:54:11.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:11 smithi079 ceph-mon[25794]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T02:54:11.810 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:11 smithi079 ceph-mon[25794]: Cluster is now healthy 2024-03-12T02:54:11.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:11 smithi115 ceph-mon[29819]: pgmap v104: 129 pgs: 9 creating+peering, 97 active+clean, 23 unknown; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T02:54:11.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:11 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:11.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:11 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:11.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:11 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:11.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:11 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/3560754543' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:11.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:11 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/3560754543' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T02:54:11.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:11 smithi115 ceph-mon[29819]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T02:54:11.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:11 smithi115 ceph-mon[29819]: Cluster is now healthy 2024-03-12T02:54:11.876 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:54:11.876 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-12T02:54:11.876 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-12T02:54:11.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:11 smithi129 ceph-mon[30321]: pgmap v104: 129 pgs: 9 creating+peering, 97 active+clean, 23 unknown; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T02:54:11.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:11 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:11.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:11 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:11.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:11 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:11.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:11 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/3560754543' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:11.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:11 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/3560754543' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T02:54:11.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:11 smithi129 ceph-mon[30321]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T02:54:11.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:11 smithi129 ceph-mon[30321]: Cluster is now healthy 2024-03-12T02:54:11.919 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T02:54:12.155 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T02:54:12.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:12 smithi115 ceph-mon[29819]: Deploying daemon prometheus.a on smithi115 2024-03-12T02:54:12.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:12 smithi079 ceph-mon[25794]: Deploying daemon prometheus.a on smithi115 2024-03-12T02:54:12.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:12 smithi129 ceph-mon[30321]: Deploying daemon prometheus.a on smithi115 2024-03-12T02:54:13.801 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:13 smithi115 ceph-mon[29819]: pgmap v105: 129 pgs: 119 active+clean, 10 unknown; 580 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 43 KiB/s rd, 2.8 KiB/s wr, 99 op/s 2024-03-12T02:54:13.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:13 smithi079 ceph-mon[25794]: pgmap v105: 129 pgs: 119 active+clean, 10 unknown; 580 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 43 KiB/s rd, 2.8 KiB/s wr, 99 op/s 2024-03-12T02:54:13.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:13 smithi129 ceph-mon[30321]: pgmap v105: 129 pgs: 119 active+clean, 10 unknown; 580 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 43 KiB/s rd, 2.8 KiB/s wr, 99 op/s 2024-03-12T02:54:14.795 INFO:teuthology.orchestra.run.smithi115.stdout:[client.1] 2024-03-12T02:54:14.795 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDWw+9lPe7xLhAAhgNcagGOAgUB+18R5ycz4Q== 2024-03-12T02:54:15.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:15 smithi115 ceph-mon[29819]: pgmap v106: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 80 KiB/s rd, 3.8 KiB/s wr, 180 op/s 2024-03-12T02:54:15.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:15 smithi115 ceph-mon[29819]: from='client.? 172.21.15.115:0/585922320' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:15.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:15 smithi115 ceph-mon[29819]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:15.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:15 smithi115 ceph-mon[29819]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T02:54:15.777 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:54:15.777 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-12T02:54:15.777 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-12T02:54:15.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:15 smithi079 ceph-mon[25794]: pgmap v106: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 80 KiB/s rd, 3.8 KiB/s wr, 180 op/s 2024-03-12T02:54:15.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:15 smithi079 ceph-mon[25794]: from='client.? 172.21.15.115:0/585922320' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:15.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:15 smithi079 ceph-mon[25794]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:15.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:15 smithi079 ceph-mon[25794]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T02:54:15.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T02:54:15.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:15 smithi129 ceph-mon[30321]: pgmap v106: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 80 KiB/s rd, 3.8 KiB/s wr, 180 op/s 2024-03-12T02:54:15.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:15 smithi129 ceph-mon[30321]: from='client.? 172.21.15.115:0/585922320' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:15.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:15 smithi129 ceph-mon[30321]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:15.852 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:15 smithi129 ceph-mon[30321]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T02:54:16.069 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T02:54:17.568 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:17 smithi129 ceph-mon[30321]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.4 KiB/s wr, 200 op/s 2024-03-12T02:54:17.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:17 smithi079 ceph-mon[25794]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.4 KiB/s wr, 200 op/s 2024-03-12T02:54:17.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:17 smithi115 ceph-mon[29819]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.4 KiB/s wr, 200 op/s 2024-03-12T02:54:17.895 INFO:teuthology.orchestra.run.smithi129.stdout:[client.2] 2024-03-12T02:54:17.895 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDZw+9l8iQFNRAAGfXyZ8v5/giPsPPtaBUeNg== 2024-03-12T02:54:18.549 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:54:18.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-12T02:54:18.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-12T02:54:18.633 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-12T02:54:18.633 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-12T02:54:18.633 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-12T02:54:18.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:18 smithi079 ceph-mon[25794]: from='client.? 172.21.15.129:0/219991041' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:18.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:18 smithi079 ceph-mon[25794]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:18.809 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:18 smithi079 ceph-mon[25794]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T02:54:18.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:18 smithi115 ceph-mon[29819]: from='client.? 172.21.15.129:0/219991041' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:18.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:18 smithi115 ceph-mon[29819]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:18.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:18 smithi115 ceph-mon[29819]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T02:54:18.875 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:18.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:18 smithi129 ceph-mon[30321]: from='client.? 172.21.15.129:0/219991041' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:18.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:18 smithi129 ceph-mon[30321]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T02:54:18.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:18 smithi129 ceph-mon[30321]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T02:54:19.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:19 smithi079 ceph-mon[25794]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 79 KiB/s rd, 3.9 KiB/s wr, 176 op/s 2024-03-12T02:54:19.688 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:19 smithi115 ceph-mon[29819]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 79 KiB/s rd, 3.9 KiB/s wr, 176 op/s 2024-03-12T02:54:19.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:19 smithi129 ceph-mon[30321]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 79 KiB/s rd, 3.9 KiB/s wr, 176 op/s 2024-03-12T02:54:20.728 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:54:21.369 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":1006526151},{"type":"v1","addr":"172.21.15.79:6801","nonce":1006526151}]},"active_addr":"172.21.15.79:6801/1006526151","active_change":"2024-03-12T02:51:24.282967+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.79:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3716737142}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":4242302406}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3988183809}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":253389884}]}]} 2024-03-12T02:54:21.374 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-12T02:54:21.375 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-12T02:54:21.375 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T02:54:21.610 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:21.716 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:21 smithi115 ceph-mon[29819]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 71 KiB/s rd, 3.5 KiB/s wr, 159 op/s 2024-03-12T02:54:21.716 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:21 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1799130266' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T02:54:21.730 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:21 smithi079 ceph-mon[25794]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 71 KiB/s rd, 3.5 KiB/s wr, 159 op/s 2024-03-12T02:54:21.730 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:21 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1799130266' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T02:54:21.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:21 smithi129 ceph-mon[30321]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 71 KiB/s rd, 3.5 KiB/s wr, 159 op/s 2024-03-12T02:54:21.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:21 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1799130266' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T02:54:22.350 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 systemd[1]: Starting Ceph prometheus.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:54:22.713 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 podman[40758]: 2024-03-12 02:54:22.364244264 +0000 UTC m=+0.018984804 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-12T02:54:22.713 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 podman[40758]: 2024-03-12 02:54:22.467330844 +0000 UTC m=+0.122071367 container create 85e505665ae840b46cdcf4f7476ef67c2a5944e4ac5f1f5a842caaa2be7cac43 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T02:54:22.713 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 podman[40758]: 2024-03-12 02:54:22.711281517 +0000 UTC m=+0.366022037 container init 85e505665ae840b46cdcf4f7476ef67c2a5944e4ac5f1f5a842caaa2be7cac43 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T02:54:23.100 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 podman[40758]: 2024-03-12 02:54:22.716533042 +0000 UTC m=+0.371273562 container start 85e505665ae840b46cdcf4f7476ef67c2a5944e4ac5f1f5a842caaa2be7cac43 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T02:54:23.100 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.740Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-12T02:54:23.100 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.740Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-12T02:54:23.100 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.740Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi115 (none))" 2024-03-12T02:54:23.100 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.740Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T02:54:23.100 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.740Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.742Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.742Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.743Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.744Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.745Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.745Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.477µs 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.745Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.746Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.746Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=14.942µs wal_replay_duration=997.479µs wbl_replay_duration=191ns total_replay_duration=1.029156ms 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.747Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.747Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-12T02:54:23.101 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.747Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T02:54:23.102 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.768Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.760546ms db_storage=714ns remote_storage=1.396µs web_handler=359ns query_engine=642ns scrape=475.526µs scrape_sd=76.037µs notify=908ns notify_sd=2.289µs rules=19.946028ms tracing=5.451µs 2024-03-12T02:54:23.102 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.768Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-12T02:54:23.102 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:54:22.768Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-12T02:54:23.102 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 bash[40758]: 85e505665ae840b46cdcf4f7476ef67c2a5944e4ac5f1f5a842caaa2be7cac43 2024-03-12T02:54:23.102 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:54:22 smithi115 systemd[1]: Started Ceph prometheus.a for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:54:23.347 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:54:23.347 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":31,"fsid":"0b28b95e-e01b-11ee-95c8-87774f69a715","created":"2024-03-12T02:50:32.098344+0000","modified":"2024-03-12T02:54:10.246117+0000","last_up_change":"2024-03-12T02:53:49.215820+0000","last_in_change":"2024-03-12T02:53:31.439447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T02:53:50.612844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-12T02:54:01.582605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-12T02:54:03.513103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-12T02:54:05.475789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-12T02:54:07.539361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fca3404-91d7-4899-8362-9b1c2a212438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6803","nonce":388305361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6805","nonce":388305361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6809","nonce":388305361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6807","nonce":388305361}]},"public_addr":"172.21.15.79:6803/388305361","cluster_addr":"172.21.15.79:6805/388305361","heartbeat_back_addr":"172.21.15.79:6809/388305361","heartbeat_front_addr":"172.21.15.79:6807/388305361","state":["exists","up"]},{"osd":1,"uuid":"79286417-821f-4092-a811-f3eaea1907bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6801","nonce":2621660585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6803","nonce":2621660585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6807","nonce":2621660585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6805","nonce":2621660585}]},"public_addr":"172.21.15.115:6801/2621660585","cluster_addr":"172.21.15.115:6803/2621660585","heartbeat_back_addr":"172.21.15.115:6807/2621660585","heartbeat_front_addr":"172.21.15.115:6805/2621660585","state":["exists","up"]},{"osd":2,"uuid":"354aa3b0-d113-4978-b0db-614c813242d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6801","nonce":2648140063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6803","nonce":2648140063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6807","nonce":2648140063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6805","nonce":2648140063}]},"public_addr":"172.21.15.129:6801/2648140063","cluster_addr":"172.21.15.129:6803/2648140063","heartbeat_back_addr":"172.21.15.129:6807/2648140063","heartbeat_front_addr":"172.21.15.129:6805/2648140063","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T02:52:59.965800+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T02:53:24.223303+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T02:53:48.054301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1172559344":"2024-03-13T02:51:24.282856+0000","172.21.15.79:0/3747240":"2024-03-13T02:51:24.282856+0000","172.21.15.79:6800/2289827629":"2024-03-13T02:51:24.282856+0000","172.21.15.79:6801/2289827629":"2024-03-13T02:51:24.282856+0000","172.21.15.79:0/2417320756":"2024-03-13T02:50:58.528795+0000","172.21.15.79:0/1664099364":"2024-03-13T02:51:24.282856+0000","172.21.15.79:6800/1227173847":"2024-03-13T02:50:58.528795+0000","172.21.15.79:0/2524931850":"2024-03-13T02:50:58.528795+0000","172.21.15.79:6801/1227173847":"2024-03-13T02:50:58.528795+0000","172.21.15.79:0/3251309801":"2024-03-13T02:50:58.528795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T02:54:23.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:23 smithi079 ceph-mon[25794]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.9 KiB/s wr, 133 op/s 2024-03-12T02:54:23.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:23.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:23.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:23.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:23 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T02:54:23.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:23 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1661377162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T02:54:23.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:23 smithi115 ceph-mon[29819]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.9 KiB/s wr, 133 op/s 2024-03-12T02:54:23.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:23 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:23.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:23 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:23.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:23 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:23.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:23 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T02:54:23.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:23 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1661377162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T02:54:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:23 smithi129 ceph-mon[30321]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.9 KiB/s wr, 133 op/s 2024-03-12T02:54:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' 2024-03-12T02:54:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:23 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T02:54:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:23 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1661377162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T02:54:24.042 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-12T02:54:24.042 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T02:54:24.289 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:23 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ignoring --setuser ceph since I am not root 2024-03-12T02:54:24.289 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:23 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ignoring --setgroup ceph since I am not root 2024-03-12T02:54:24.289 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:24.349 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: ignoring --setuser ceph since I am not root 2024-03-12T02:54:24.349 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: ignoring --setgroup ceph since I am not root 2024-03-12T02:54:24.350 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:23 smithi115 ceph-mgr[31336]: -- 172.21.15.115:0/2793953553 <== mon.0 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558963f165a0 con 0x558963eeac00 2024-03-12T02:54:24.378 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:23 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: ignoring --setuser ceph since I am not root 2024-03-12T02:54:24.378 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:23 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: ignoring --setgroup ceph since I am not root 2024-03-12T02:54:24.378 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:23 smithi129 ceph-mgr[32397]: -- 172.21.15.129:0/2861418959 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5616185a05a0 con 0x561618575000 2024-03-12T02:54:24.559 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:24 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:24.508+0000 7fb097305200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T02:54:24.849 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:24 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:24.467+0000 7fd9a0cea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T02:54:24.849 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:24 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:24.560+0000 7fd9a0cea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T02:54:24.878 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:24 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:24.469+0000 7f84f1669200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T02:54:24.878 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:24 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:24.561+0000 7f84f1669200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T02:54:24.906 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:24 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:24.610+0000 7fb097305200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T02:54:25.192 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:24 smithi115 ceph-mon[29819]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T02:54:25.192 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:24 smithi115 ceph-mon[29819]: mgrmap e17: y(active, since 2m), standbys: z, x 2024-03-12T02:54:25.194 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:24 smithi129 ceph-mon[30321]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T02:54:25.194 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:24 smithi129 ceph-mon[30321]: mgrmap e17: y(active, since 2m), standbys: z, x 2024-03-12T02:54:25.253 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:24 smithi079 ceph-mon[25794]: from='mgr.14152 172.21.15.79:0/1153814625' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T02:54:25.254 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:24 smithi079 ceph-mon[25794]: mgrmap e17: y(active, since 2m), standbys: z, x 2024-03-12T02:54:25.520 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:25 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:25.191+0000 7fd9a0cea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T02:54:25.521 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:25 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:25.284+0000 7fd9a0cea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T02:54:25.521 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:25 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:25.425+0000 7fd9a0cea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T02:54:25.521 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:25 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:25.519+0000 7fd9a0cea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T02:54:25.522 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:25 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:25.280+0000 7fb097305200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T02:54:25.523 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:25 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:25.373+0000 7fb097305200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T02:54:25.523 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:25 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:25.521+0000 7fb097305200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T02:54:25.524 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:25 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:25.191+0000 7f84f1669200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T02:54:25.524 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:25 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:25.283+0000 7f84f1669200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T02:54:25.524 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:25 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:25.426+0000 7f84f1669200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T02:54:25.524 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:25 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:25.522+0000 7f84f1669200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T02:54:25.809 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:25 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:25.617+0000 7fb097305200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T02:54:25.810 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:25 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:25.701+0000 7fb097305200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T02:54:25.849 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:25 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:25.601+0000 7fd9a0cea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T02:54:25.867 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:25 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:25.603+0000 7f84f1669200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T02:54:25.867 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:25 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:25.864+0000 7f84f1669200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T02:54:26.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:25 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:25.975+0000 7f84f1669200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T02:54:26.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:26 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:26.060+0000 7f84f1669200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T02:54:26.149 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:25 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:25.858+0000 7fd9a0cea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T02:54:26.149 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:25 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:25.968+0000 7fd9a0cea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T02:54:26.149 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:26 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:26.053+0000 7fd9a0cea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T02:54:26.149 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:26 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:26.146+0000 7fd9a0cea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T02:54:26.175 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:54:26.175 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":31,"fsid":"0b28b95e-e01b-11ee-95c8-87774f69a715","created":"2024-03-12T02:50:32.098344+0000","modified":"2024-03-12T02:54:10.246117+0000","last_up_change":"2024-03-12T02:53:49.215820+0000","last_in_change":"2024-03-12T02:53:31.439447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T02:53:50.612844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-12T02:54:01.582605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-12T02:54:03.513103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-12T02:54:05.475789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-12T02:54:07.539361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fca3404-91d7-4899-8362-9b1c2a212438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6803","nonce":388305361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6805","nonce":388305361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6809","nonce":388305361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":388305361},{"type":"v1","addr":"172.21.15.79:6807","nonce":388305361}]},"public_addr":"172.21.15.79:6803/388305361","cluster_addr":"172.21.15.79:6805/388305361","heartbeat_back_addr":"172.21.15.79:6809/388305361","heartbeat_front_addr":"172.21.15.79:6807/388305361","state":["exists","up"]},{"osd":1,"uuid":"79286417-821f-4092-a811-f3eaea1907bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6801","nonce":2621660585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6803","nonce":2621660585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6807","nonce":2621660585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2621660585},{"type":"v1","addr":"172.21.15.115:6805","nonce":2621660585}]},"public_addr":"172.21.15.115:6801/2621660585","cluster_addr":"172.21.15.115:6803/2621660585","heartbeat_back_addr":"172.21.15.115:6807/2621660585","heartbeat_front_addr":"172.21.15.115:6805/2621660585","state":["exists","up"]},{"osd":2,"uuid":"354aa3b0-d113-4978-b0db-614c813242d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6801","nonce":2648140063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6803","nonce":2648140063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6807","nonce":2648140063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2648140063},{"type":"v1","addr":"172.21.15.129:6805","nonce":2648140063}]},"public_addr":"172.21.15.129:6801/2648140063","cluster_addr":"172.21.15.129:6803/2648140063","heartbeat_back_addr":"172.21.15.129:6807/2648140063","heartbeat_front_addr":"172.21.15.129:6805/2648140063","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T02:52:59.965800+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T02:53:24.223303+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T02:53:48.054301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1172559344":"2024-03-13T02:51:24.282856+0000","172.21.15.79:0/3747240":"2024-03-13T02:51:24.282856+0000","172.21.15.79:6800/2289827629":"2024-03-13T02:51:24.282856+0000","172.21.15.79:6801/2289827629":"2024-03-13T02:51:24.282856+0000","172.21.15.79:0/2417320756":"2024-03-13T02:50:58.528795+0000","172.21.15.79:0/1664099364":"2024-03-13T02:51:24.282856+0000","172.21.15.79:6800/1227173847":"2024-03-13T02:50:58.528795+0000","172.21.15.79:0/2524931850":"2024-03-13T02:50:58.528795+0000","172.21.15.79:6801/1227173847":"2024-03-13T02:50:58.528795+0000","172.21.15.79:0/3251309801":"2024-03-13T02:50:58.528795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T02:54:26.175 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:25 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:25.975+0000 7fb097305200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T02:54:26.175 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:26 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:26.087+0000 7fb097305200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T02:54:26.455 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:26 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1533915869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T02:54:26.455 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:26 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:26.174+0000 7fb097305200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T02:54:26.455 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:26 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:26.274+0000 7fb097305200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T02:54:26.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:26 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1533915869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T02:54:26.600 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:26 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:26.320+0000 7fd9a0cea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T02:54:26.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:26 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1533915869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T02:54:26.628 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:26 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:26.152+0000 7f84f1669200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T02:54:26.628 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:26 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:26.329+0000 7f84f1669200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T02:54:26.767 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:26 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:26.452+0000 7fb097305200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T02:54:26.799 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-12T02:54:26.800 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-12T02:54:26.800 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-12T02:54:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:27.102 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:27.110 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:27.599 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:27 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:27.269+0000 7fd9a0cea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T02:54:27.600 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:27 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:27.398+0000 7fd9a0cea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T02:54:27.628 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:27 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:27.283+0000 7f84f1669200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T02:54:27.686 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:27 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:27.414+0000 7f84f1669200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T02:54:27.739 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:27 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:27.474+0000 7fb097305200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T02:54:27.739 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:27 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:27.609+0000 7fb097305200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T02:54:28.011 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:27 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:27.728+0000 7fd9a0cea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T02:54:28.012 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:27 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:27.817+0000 7fd9a0cea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T02:54:28.012 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:28 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:28.009+0000 7fd9a0cea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:54:28.036 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:27 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:27.749+0000 7f84f1669200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T02:54:28.037 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:27 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:27.839+0000 7f84f1669200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T02:54:28.037 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:28 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:28.034+0000 7f84f1669200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:54:28.041 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:27 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:27.956+0000 7fb097305200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T02:54:28.309 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:28 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:28.053+0000 7fb097305200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T02:54:28.310 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:28 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:28.250+0000 7fb097305200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:54:28.349 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:28 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:28.192+0000 7fd9a0cea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T02:54:28.377 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:28 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:28.218+0000 7f84f1669200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T02:54:28.678 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:28 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:28.435+0000 7fb097305200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T02:54:28.732 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:28 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:28.474+0000 7fd9a0cea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:54:28.766 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:28 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:28.504+0000 7f84f1669200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:54:28.766 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:28 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:28.764+0000 7f84f1669200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T02:54:28.988 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:28 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:28.726+0000 7fb097305200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T02:54:28.989 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:28 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:28.986+0000 7fb097305200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T02:54:29.099 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:28 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:28.729+0000 7fd9a0cea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T02:54:29.099 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:28 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:28.960+0000 7fd9a0cea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T02:54:29.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:29 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:28.999+0000 7f84f1669200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T02:54:29.309 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:29 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:29.227+0000 7fb097305200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T02:54:30.099 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:29 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:29.758+0000 7fd9a0cea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T02:54:30.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:29 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:29.795+0000 7f84f1669200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T02:54:30.309 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:30.062+0000 7fb097305200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T02:54:30.599 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:30.326+0000 7fd9a0cea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T02:54:30.600 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:30.424+0000 7fd9a0cea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T02:54:30.630 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:30.367+0000 7f84f1669200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T02:54:30.630 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:30.466+0000 7f84f1669200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T02:54:30.985 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:30.660+0000 7fb097305200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T02:54:30.985 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:30.763+0000 7fb097305200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T02:54:30.985 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:30.982+0000 7fb097305200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: Standby manager daemon x restarted 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: Standby manager daemon x started 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: Standby manager daemon z restarted 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: Standby manager daemon z started 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T02:54:30.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:30 smithi079 ceph-mon[25794]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:54:31.099 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:30.640+0000 7fd9a0cea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: [12/Mar/2024:02:54:30] ENGINE Bus STARTING 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: CherryPy Checker: 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: The Application mounted at '' has an empty config. 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: [12/Mar/2024:02:54:30] ENGINE Serving on http://:::9283 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x[31314]: [12/Mar/2024:02:54:30] ENGINE Bus STARTED 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: Standby manager daemon x restarted 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: Standby manager daemon x started 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:54:31.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T02:54:31.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:54:31.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: Standby manager daemon z restarted 2024-03-12T02:54:31.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: Standby manager daemon z started 2024-03-12T02:54:31.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T02:54:31.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:54:31.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T02:54:31.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:30 smithi115 ceph-mon[29819]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:54:31.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:30.682+0000 7f84f1669200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T02:54:31.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: [12/Mar/2024:02:54:30] ENGINE Bus STARTING 2024-03-12T02:54:31.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: CherryPy Checker: 2024-03-12T02:54:31.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: The Application mounted at '' has an empty config. 2024-03-12T02:54:31.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: 2024-03-12T02:54:31.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: [12/Mar/2024:02:54:30] ENGINE Serving on http://:::9283 2024-03-12T02:54:31.128 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z[32342]: [12/Mar/2024:02:54:30] ENGINE Bus STARTED 2024-03-12T02:54:31.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: Standby manager daemon x restarted 2024-03-12T02:54:31.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: Standby manager daemon x started 2024-03-12T02:54:31.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T02:54:31.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:54:31.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T02:54:31.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.115:0/1488466657' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:54:31.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: Standby manager daemon z restarted 2024-03-12T02:54:31.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: Standby manager daemon z started 2024-03-12T02:54:31.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T02:54:31.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T02:54:31.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T02:54:31.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:30 smithi129 ceph-mon[30321]: from='mgr.? 172.21.15.129:0/2541236822' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T02:54:31.239 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:54:31] ENGINE Bus STARTING 2024-03-12T02:54:31.239 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: CherryPy Checker: 2024-03-12T02:54:31.239 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: The Application mounted at '' has an empty config. 2024-03-12T02:54:31.239 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:54:31.239 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:54:31] ENGINE Serving on http://:::9283 2024-03-12T02:54:31.239 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:54:31] ENGINE Bus STARTED 2024-03-12T02:54:31.917 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-12T02:54:31.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: Active manager daemon y restarted 2024-03-12T02:54:31.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: Activating manager daemon y 2024-03-12T02:54:31.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: osdmap e32: 3 total, 3 up, 3 in 2024-03-12T02:54:31.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: mgrmap e19: y(active, starting, since 0.0102438s), standbys: x, z 2024-03-12T02:54:31.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:54:31.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:54:31.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:54:31.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T02:54:31.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: Manager daemon y is now available 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:31.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:54:31.920 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T02:54:31.920 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:31 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T02:54:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-12T02:54:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: Active manager daemon y restarted 2024-03-12T02:54:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: Activating manager daemon y 2024-03-12T02:54:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: osdmap e32: 3 total, 3 up, 3 in 2024-03-12T02:54:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: mgrmap e19: y(active, starting, since 0.0102438s), standbys: x, z 2024-03-12T02:54:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:54:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:54:32.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: Manager daemon y is now available 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:54:32.101 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T02:54:32.102 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:31 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T02:54:32.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-12T02:54:32.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: Active manager daemon y restarted 2024-03-12T02:54:32.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: Activating manager daemon y 2024-03-12T02:54:32.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: osdmap e32: 3 total, 3 up, 3 in 2024-03-12T02:54:32.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: mgrmap e19: y(active, starting, since 0.0102438s), standbys: x, z 2024-03-12T02:54:32.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T02:54:32.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T02:54:32.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T02:54:32.129 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T02:54:32.130 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T02:54:32.130 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T02:54:32.130 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T02:54:32.130 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T02:54:32.130 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T02:54:32.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T02:54:32.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T02:54:32.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T02:54:32.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: Manager daemon y is now available 2024-03-12T02:54:32.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:54:32.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T02:54:32.132 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:32.132 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:54:32.132 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T02:54:32.132 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:31 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T02:54:32.261 INFO:teuthology.orchestra.run.smithi079.stdout:34359738387 2024-03-12T02:54:32.262 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-12T02:54:32.689 INFO:teuthology.orchestra.run.smithi079.stdout:55834574863 2024-03-12T02:54:32.690 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-12T02:54:33.007 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:32 smithi079 ceph-mon[25794]: [12/Mar/2024:02:54:31] ENGINE Bus STARTING 2024-03-12T02:54:33.007 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:32 smithi079 ceph-mon[25794]: [12/Mar/2024:02:54:31] ENGINE Serving on http://172.21.15.79:8765 2024-03-12T02:54:33.007 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:32 smithi079 ceph-mon[25794]: [12/Mar/2024:02:54:31] ENGINE Serving on https://172.21.15.79:7150 2024-03-12T02:54:33.007 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:32 smithi079 ceph-mon[25794]: [12/Mar/2024:02:54:31] ENGINE Bus STARTED 2024-03-12T02:54:33.007 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:32 smithi079 ceph-mon[25794]: mgrmap e20: y(active, since 1.01502s), standbys: x, z 2024-03-12T02:54:33.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:32 smithi115 ceph-mon[29819]: [12/Mar/2024:02:54:31] ENGINE Bus STARTING 2024-03-12T02:54:33.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:32 smithi115 ceph-mon[29819]: [12/Mar/2024:02:54:31] ENGINE Serving on http://172.21.15.79:8765 2024-03-12T02:54:33.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:32 smithi115 ceph-mon[29819]: [12/Mar/2024:02:54:31] ENGINE Serving on https://172.21.15.79:7150 2024-03-12T02:54:33.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:32 smithi115 ceph-mon[29819]: [12/Mar/2024:02:54:31] ENGINE Bus STARTED 2024-03-12T02:54:33.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:32 smithi115 ceph-mon[29819]: mgrmap e20: y(active, since 1.01502s), standbys: x, z 2024-03-12T02:54:33.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:32 smithi129 ceph-mon[30321]: [12/Mar/2024:02:54:31] ENGINE Bus STARTING 2024-03-12T02:54:33.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:32 smithi129 ceph-mon[30321]: [12/Mar/2024:02:54:31] ENGINE Serving on http://172.21.15.79:8765 2024-03-12T02:54:33.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:32 smithi129 ceph-mon[30321]: [12/Mar/2024:02:54:31] ENGINE Serving on https://172.21.15.79:7150 2024-03-12T02:54:33.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:32 smithi129 ceph-mon[30321]: [12/Mar/2024:02:54:31] ENGINE Bus STARTED 2024-03-12T02:54:33.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:32 smithi129 ceph-mon[30321]: mgrmap e20: y(active, since 1.01502s), standbys: x, z 2024-03-12T02:54:33.331 INFO:teuthology.orchestra.run.smithi079.stdout:77309411337 2024-03-12T02:54:33.331 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-12T02:54:33.385 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:33.432 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:33.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:33 smithi079 ceph-mon[25794]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:33.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:33 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:33.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:33 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:34.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:33 smithi115 ceph-mon[29819]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:34.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:33 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:34.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:33 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:34.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:33 smithi129 ceph-mon[30321]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:34.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:33 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:34.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:33 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:35.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:34 smithi079 ceph-mon[25794]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:35.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:34 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:35.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:34 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:35.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:34 smithi079 ceph-mon[25794]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-12T02:54:35.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:34 smithi115 ceph-mon[29819]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:35.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:34 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:35.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:34 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:35.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:34 smithi115 ceph-mon[29819]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-12T02:54:35.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:34 smithi129 ceph-mon[30321]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:35.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:34 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:35.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:34 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:35.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:34 smithi129 ceph-mon[30321]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-12T02:54:35.801 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:36.180 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:36 smithi079 ceph-mon[25794]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:36.180 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:36 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:36.181 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:36 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:36.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:36 smithi115 ceph-mon[29819]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:36.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:36 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:36.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:36 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:36.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:36 smithi129 ceph-mon[30321]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:36.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:36 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:36.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:36 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:36.429 INFO:teuthology.orchestra.run.smithi079.stdout:34359738388 2024-03-12T02:54:37.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:37 smithi079 ceph-mon[25794]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-03-12T02:54:37.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:37 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:37.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:37 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:37.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:37 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:54:37.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:37 smithi079 ceph-mon[25794]: Adjusting osd_memory_target on smithi115 to 16064M 2024-03-12T02:54:37.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:37 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:37.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:37 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/1066586896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T02:54:37.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:37 smithi115 ceph-mon[29819]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-03-12T02:54:37.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:37 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:37.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:37 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:37.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:37 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:54:37.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:37 smithi115 ceph-mon[29819]: Adjusting osd_memory_target on smithi115 to 16064M 2024-03-12T02:54:37.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:37 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:37.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:37 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/1066586896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T02:54:37.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:37 smithi129 ceph-mon[30321]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-03-12T02:54:37.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:37 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:37.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:37 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:37.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:37 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:54:37.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:37 smithi129 ceph-mon[30321]: Adjusting osd_memory_target on smithi115 to 16064M 2024-03-12T02:54:37.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:37 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:37.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:37 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/1066586896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T02:54:37.391 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738387 got 34359738388 for osd.0 2024-03-12T02:54:37.391 DEBUG:teuthology.parallel:result is None 2024-03-12T02:54:38.947 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:38 smithi079 ceph-mon[25794]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-12T02:54:38.948 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:38 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:38.948 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:38 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:39.090 INFO:teuthology.orchestra.run.smithi079.stdout:55834574865 2024-03-12T02:54:39.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:38 smithi115 ceph-mon[29819]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-12T02:54:39.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:38 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:39.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:38 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:39.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:38 smithi129 ceph-mon[30321]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-12T02:54:39.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:38 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:39.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:38 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:39.304 INFO:teuthology.orchestra.run.smithi079.stdout:77309411339 2024-03-12T02:54:40.068 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411337 got 77309411339 for osd.2 2024-03-12T02:54:40.068 DEBUG:teuthology.parallel:result is None 2024-03-12T02:54:40.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-12T02:54:40.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:40.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Adjusting osd_memory_target on smithi079 to 16064M 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi079:/etc/ceph/ceph.conf 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2659107890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/2185583161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:54:40.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:54:40.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:54:40.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:54:40.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:54:40.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:54:40.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:54:40.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-mon[25794]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:54:40.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-12T02:54:40.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:40.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:40.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:54:40.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Adjusting osd_memory_target on smithi079 to 16064M 2024-03-12T02:54:40.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:40.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:54:40.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:54:40.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi079:/etc/ceph/ceph.conf 2024-03-12T02:54:40.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-12T02:54:40.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2659107890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/2185583161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:54:40.351 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:40 smithi115 ceph-mon[29819]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:54:40.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-12T02:54:40.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:40.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:40.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T02:54:40.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Adjusting osd_memory_target on smithi079 to 16064M 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi079:/etc/ceph/ceph.conf 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2659107890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/2185583161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:54:40.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-12T02:54:40.380 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:54:40.380 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:54:40.380 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:40 smithi129 ceph-mon[30321]: Updating smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T02:54:40.572 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574863 got 55834574865 for osd.1 2024-03-12T02:54:40.573 DEBUG:teuthology.parallel:result is None 2024-03-12T02:54:40.573 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-12T02:54:40.573 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T02:54:40.818 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:41.056 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:40 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ::ffff:172.21.15.115 - - [12/Mar/2024:02:54:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T02:54:41.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:41 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:41 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:41 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:41 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:41 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:41 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:41 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:41 smithi079 ceph-mon[25794]: Deploying daemon alertmanager.a on smithi079 2024-03-12T02:54:41.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:41 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:41 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:41 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:41 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:41 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:41 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:41 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:41 smithi129 ceph-mon[30321]: Deploying daemon alertmanager.a on smithi079 2024-03-12T02:54:41.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:41 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:41 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:41 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:41 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:41 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:41 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:41 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:41.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:41 smithi115 ceph-mon[29819]: Deploying daemon alertmanager.a on smithi079 2024-03-12T02:54:42.403 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:42 smithi079 ceph-mon[25794]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T02:54:42.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:42 smithi115 ceph-mon[29819]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T02:54:42.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:42 smithi129 ceph-mon[30321]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T02:54:43.197 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:54:43.199 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-03-12T02:54:44.215 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-12T02:54:43.014118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595984,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96636,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126532,"statfs":{"total":287972524032,"available":287873568768,"internally_reserved":0,"allocated":3182592,"data_stored":2109726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.001259"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004573+0000","last_change":"2024-03-12T02:54:09.262852+0000","last_active":"2024-03-12T02:54:31.004573+0000","last_peered":"2024-03-12T02:54:31.004573+0000","last_clean":"2024-03-12T02:54:31.004573+0000","last_became_active":"2024-03-12T02:54:09.262475+0000","last_became_peered":"2024-03-12T02:54:09.262475+0000","last_unstale":"2024-03-12T02:54:31.004573+0000","last_undegraded":"2024-03-12T02:54:31.004573+0000","last_fullsized":"2024-03-12T02:54:31.004573+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:05:03.570464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006201+0000","last_change":"2024-03-12T02:54:07.256190+0000","last_active":"2024-03-12T02:54:31.006201+0000","last_peered":"2024-03-12T02:54:31.006201+0000","last_clean":"2024-03-12T02:54:31.006201+0000","last_became_active":"2024-03-12T02:54:07.255745+0000","last_became_peered":"2024-03-12T02:54:07.255745+0000","last_unstale":"2024-03-12T02:54:31.006201+0000","last_undegraded":"2024-03-12T02:54:31.006201+0000","last_fullsized":"2024-03-12T02:54:31.006201+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:15:33.801516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005851+0000","last_change":"2024-03-12T02:54:03.226988+0000","last_active":"2024-03-12T02:54:31.005851+0000","last_peered":"2024-03-12T02:54:31.005851+0000","last_clean":"2024-03-12T02:54:31.005851+0000","last_became_active":"2024-03-12T02:54:03.226722+0000","last_became_peered":"2024-03-12T02:54:03.226722+0000","last_unstale":"2024-03-12T02:54:31.005851+0000","last_undegraded":"2024-03-12T02:54:31.005851+0000","last_fullsized":"2024-03-12T02:54:31.005851+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:31:19.357681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'16","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005035+0000","last_change":"2024-03-12T02:54:05.235077+0000","last_active":"2024-03-12T02:54:31.005035+0000","last_peered":"2024-03-12T02:54:31.005035+0000","last_clean":"2024-03-12T02:54:31.005035+0000","last_became_active":"2024-03-12T02:54:05.233185+0000","last_became_peered":"2024-03-12T02:54:05.233185+0000","last_unstale":"2024-03-12T02:54:31.005035+0000","last_undegraded":"2024-03-12T02:54:31.005035+0000","last_fullsized":"2024-03-12T02:54:31.005035+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:10:08.885568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002691+0000","last_change":"2024-03-12T02:54:09.261727+0000","last_active":"2024-03-12T02:54:31.002691+0000","last_peered":"2024-03-12T02:54:31.002691+0000","last_clean":"2024-03-12T02:54:31.002691+0000","last_became_active":"2024-03-12T02:54:09.261355+0000","last_became_peered":"2024-03-12T02:54:09.261355+0000","last_unstale":"2024-03-12T02:54:31.002691+0000","last_undegraded":"2024-03-12T02:54:31.002691+0000","last_fullsized":"2024-03-12T02:54:31.002691+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:47.986881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004407+0000","last_change":"2024-03-12T02:54:07.257291+0000","last_active":"2024-03-12T02:54:31.004407+0000","last_peered":"2024-03-12T02:54:31.004407+0000","last_clean":"2024-03-12T02:54:31.004407+0000","last_became_active":"2024-03-12T02:54:07.256788+0000","last_became_peered":"2024-03-12T02:54:07.256788+0000","last_unstale":"2024-03-12T02:54:31.004407+0000","last_undegraded":"2024-03-12T02:54:31.004407+0000","last_fullsized":"2024-03-12T02:54:31.004407+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:34.754417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005320+0000","last_change":"2024-03-12T02:54:03.227646+0000","last_active":"2024-03-12T02:54:31.005320+0000","last_peered":"2024-03-12T02:54:31.005320+0000","last_clean":"2024-03-12T02:54:31.005320+0000","last_became_active":"2024-03-12T02:54:03.227360+0000","last_became_peered":"2024-03-12T02:54:03.227360+0000","last_unstale":"2024-03-12T02:54:31.005320+0000","last_undegraded":"2024-03-12T02:54:31.005320+0000","last_fullsized":"2024-03-12T02:54:31.005320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:18:25.747636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005600+0000","last_change":"2024-03-12T02:54:05.234431+0000","last_active":"2024-03-12T02:54:31.005600+0000","last_peered":"2024-03-12T02:54:31.005600+0000","last_clean":"2024-03-12T02:54:31.005600+0000","last_became_active":"2024-03-12T02:54:05.233052+0000","last_became_peered":"2024-03-12T02:54:05.233052+0000","last_unstale":"2024-03-12T02:54:31.005600+0000","last_undegraded":"2024-03-12T02:54:31.005600+0000","last_fullsized":"2024-03-12T02:54:31.005600+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:55:29.069618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006204+0000","last_change":"2024-03-12T02:54:09.261329+0000","last_active":"2024-03-12T02:54:31.006204+0000","last_peered":"2024-03-12T02:54:31.006204+0000","last_clean":"2024-03-12T02:54:31.006204+0000","last_became_active":"2024-03-12T02:54:09.261036+0000","last_became_peered":"2024-03-12T02:54:09.261036+0000","last_unstale":"2024-03-12T02:54:31.006204+0000","last_undegraded":"2024-03-12T02:54:31.006204+0000","last_fullsized":"2024-03-12T02:54:31.006204+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:15:38.555693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004561+0000","last_change":"2024-03-12T02:54:07.256265+0000","last_active":"2024-03-12T02:54:31.004561+0000","last_peered":"2024-03-12T02:54:31.004561+0000","last_clean":"2024-03-12T02:54:31.004561+0000","last_became_active":"2024-03-12T02:54:07.255716+0000","last_became_peered":"2024-03-12T02:54:07.255716+0000","last_unstale":"2024-03-12T02:54:31.004561+0000","last_undegraded":"2024-03-12T02:54:31.004561+0000","last_fullsized":"2024-03-12T02:54:31.004561+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:48:13.374889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.009809+0000","last_change":"2024-03-12T02:54:03.225600+0000","last_active":"2024-03-12T02:54:31.009809+0000","last_peered":"2024-03-12T02:54:31.009809+0000","last_clean":"2024-03-12T02:54:31.009809+0000","last_became_active":"2024-03-12T02:54:03.225188+0000","last_became_peered":"2024-03-12T02:54:03.225188+0000","last_unstale":"2024-03-12T02:54:31.009809+0000","last_undegraded":"2024-03-12T02:54:31.009809+0000","last_fullsized":"2024-03-12T02:54:31.009809+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:17:16.972224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005589+0000","last_change":"2024-03-12T02:54:05.237711+0000","last_active":"2024-03-12T02:54:31.005589+0000","last_peered":"2024-03-12T02:54:31.005589+0000","last_clean":"2024-03-12T02:54:31.005589+0000","last_became_active":"2024-03-12T02:54:05.237355+0000","last_became_peered":"2024-03-12T02:54:05.237355+0000","last_unstale":"2024-03-12T02:54:31.005589+0000","last_undegraded":"2024-03-12T02:54:31.005589+0000","last_fullsized":"2024-03-12T02:54:31.005589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:18:52.055675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004408+0000","last_change":"2024-03-12T02:54:09.256531+0000","last_active":"2024-03-12T02:54:31.004408+0000","last_peered":"2024-03-12T02:54:31.004408+0000","last_clean":"2024-03-12T02:54:31.004408+0000","last_became_active":"2024-03-12T02:54:09.256212+0000","last_became_peered":"2024-03-12T02:54:09.256212+0000","last_unstale":"2024-03-12T02:54:31.004408+0000","last_undegraded":"2024-03-12T02:54:31.004408+0000","last_fullsized":"2024-03-12T02:54:31.004408+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:27:46.690388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003871+0000","last_change":"2024-03-12T02:54:07.257143+0000","last_active":"2024-03-12T02:54:31.003871+0000","last_peered":"2024-03-12T02:54:31.003871+0000","last_clean":"2024-03-12T02:54:31.003871+0000","last_became_active":"2024-03-12T02:54:07.256713+0000","last_became_peered":"2024-03-12T02:54:07.256713+0000","last_unstale":"2024-03-12T02:54:31.003871+0000","last_undegraded":"2024-03-12T02:54:31.003871+0000","last_fullsized":"2024-03-12T02:54:31.003871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:03:45.354354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008080+0000","last_change":"2024-03-12T02:54:03.225555+0000","last_active":"2024-03-12T02:54:31.008080+0000","last_peered":"2024-03-12T02:54:31.008080+0000","last_clean":"2024-03-12T02:54:31.008080+0000","last_became_active":"2024-03-12T02:54:03.223730+0000","last_became_peered":"2024-03-12T02:54:03.223730+0000","last_unstale":"2024-03-12T02:54:31.008080+0000","last_undegraded":"2024-03-12T02:54:31.008080+0000","last_fullsized":"2024-03-12T02:54:31.008080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:48:38.603242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.009720+0000","last_change":"2024-03-12T02:54:05.236261+0000","last_active":"2024-03-12T02:54:31.009720+0000","last_peered":"2024-03-12T02:54:31.009720+0000","last_clean":"2024-03-12T02:54:31.009720+0000","last_became_active":"2024-03-12T02:54:05.235912+0000","last_became_peered":"2024-03-12T02:54:05.235912+0000","last_unstale":"2024-03-12T02:54:31.009720+0000","last_undegraded":"2024-03-12T02:54:31.009720+0000","last_fullsized":"2024-03-12T02:54:31.009720+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:32:30.495642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004989+0000","last_change":"2024-03-12T02:54:09.256454+0000","last_active":"2024-03-12T02:54:31.004989+0000","last_peered":"2024-03-12T02:54:31.004989+0000","last_clean":"2024-03-12T02:54:31.004989+0000","last_became_active":"2024-03-12T02:54:09.256084+0000","last_became_peered":"2024-03-12T02:54:09.256084+0000","last_unstale":"2024-03-12T02:54:31.004989+0000","last_undegraded":"2024-03-12T02:54:31.004989+0000","last_fullsized":"2024-03-12T02:54:31.004989+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:09:45.979440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008011+0000","last_change":"2024-03-12T02:54:07.251840+0000","last_active":"2024-03-12T02:54:31.008011+0000","last_peered":"2024-03-12T02:54:31.008011+0000","last_clean":"2024-03-12T02:54:31.008011+0000","last_became_active":"2024-03-12T02:54:07.251524+0000","last_became_peered":"2024-03-12T02:54:07.251524+0000","last_unstale":"2024-03-12T02:54:31.008011+0000","last_undegraded":"2024-03-12T02:54:31.008011+0000","last_fullsized":"2024-03-12T02:54:31.008011+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:27:15.892746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008080+0000","last_change":"2024-03-12T02:54:03.224079+0000","last_active":"2024-03-12T02:54:31.008080+0000","last_peered":"2024-03-12T02:54:31.008080+0000","last_clean":"2024-03-12T02:54:31.008080+0000","last_became_active":"2024-03-12T02:54:03.223147+0000","last_became_peered":"2024-03-12T02:54:03.223147+0000","last_unstale":"2024-03-12T02:54:31.008080+0000","last_undegraded":"2024-03-12T02:54:31.008080+0000","last_fullsized":"2024-03-12T02:54:31.008080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:05:47.427879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'23","reported_seq":65,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008158+0000","last_change":"2024-03-12T02:54:05.234670+0000","last_active":"2024-03-12T02:54:31.008158+0000","last_peered":"2024-03-12T02:54:31.008158+0000","last_clean":"2024-03-12T02:54:31.008158+0000","last_became_active":"2024-03-12T02:54:05.234310+0000","last_became_peered":"2024-03-12T02:54:05.234310+0000","last_unstale":"2024-03-12T02:54:31.008158+0000","last_undegraded":"2024-03-12T02:54:31.008158+0000","last_fullsized":"2024-03-12T02:54:31.008158+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:01:50.146305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004616+0000","last_change":"2024-03-12T02:54:09.261629+0000","last_active":"2024-03-12T02:54:31.004616+0000","last_peered":"2024-03-12T02:54:31.004616+0000","last_clean":"2024-03-12T02:54:31.004616+0000","last_became_active":"2024-03-12T02:54:09.261334+0000","last_became_peered":"2024-03-12T02:54:09.261334+0000","last_unstale":"2024-03-12T02:54:31.004616+0000","last_undegraded":"2024-03-12T02:54:31.004616+0000","last_fullsized":"2024-03-12T02:54:31.004616+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:34:56.643839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007849+0000","last_change":"2024-03-12T02:54:07.255272+0000","last_active":"2024-03-12T02:54:31.007849+0000","last_peered":"2024-03-12T02:54:31.007849+0000","last_clean":"2024-03-12T02:54:31.007849+0000","last_became_active":"2024-03-12T02:54:07.254979+0000","last_became_peered":"2024-03-12T02:54:07.254979+0000","last_unstale":"2024-03-12T02:54:31.007849+0000","last_undegraded":"2024-03-12T02:54:31.007849+0000","last_fullsized":"2024-03-12T02:54:31.007849+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:49.219851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004296+0000","last_change":"2024-03-12T02:54:03.221816+0000","last_active":"2024-03-12T02:54:31.004296+0000","last_peered":"2024-03-12T02:54:31.004296+0000","last_clean":"2024-03-12T02:54:31.004296+0000","last_became_active":"2024-03-12T02:54:03.221022+0000","last_became_peered":"2024-03-12T02:54:03.221022+0000","last_unstale":"2024-03-12T02:54:31.004296+0000","last_undegraded":"2024-03-12T02:54:31.004296+0000","last_fullsized":"2024-03-12T02:54:31.004296+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:39:50.080867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'11","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007405+0000","last_change":"2024-03-12T02:54:05.234573+0000","last_active":"2024-03-12T02:54:31.007405+0000","last_peered":"2024-03-12T02:54:31.007405+0000","last_clean":"2024-03-12T02:54:31.007405+0000","last_became_active":"2024-03-12T02:54:05.234055+0000","last_became_peered":"2024-03-12T02:54:05.234055+0000","last_unstale":"2024-03-12T02:54:31.007405+0000","last_undegraded":"2024-03-12T02:54:31.007405+0000","last_fullsized":"2024-03-12T02:54:31.007405+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:20.329923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004359+0000","last_change":"2024-03-12T02:54:09.258576+0000","last_active":"2024-03-12T02:54:31.004359+0000","last_peered":"2024-03-12T02:54:31.004359+0000","last_clean":"2024-03-12T02:54:31.004359+0000","last_became_active":"2024-03-12T02:54:09.257789+0000","last_became_peered":"2024-03-12T02:54:09.257789+0000","last_unstale":"2024-03-12T02:54:31.004359+0000","last_undegraded":"2024-03-12T02:54:31.004359+0000","last_fullsized":"2024-03-12T02:54:31.004359+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:10:02.425696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.268607+0000","last_change":"2024-03-12T02:54:07.253050+0000","last_active":"2024-03-12T02:54:33.268607+0000","last_peered":"2024-03-12T02:54:33.268607+0000","last_clean":"2024-03-12T02:54:33.268607+0000","last_became_active":"2024-03-12T02:54:07.252317+0000","last_became_peered":"2024-03-12T02:54:07.252317+0000","last_unstale":"2024-03-12T02:54:33.268607+0000","last_undegraded":"2024-03-12T02:54:33.268607+0000","last_fullsized":"2024-03-12T02:54:33.268607+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:38.947170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002857+0000","last_change":"2024-03-12T02:54:03.227812+0000","last_active":"2024-03-12T02:54:31.002857+0000","last_peered":"2024-03-12T02:54:31.002857+0000","last_clean":"2024-03-12T02:54:31.002857+0000","last_became_active":"2024-03-12T02:54:03.227429+0000","last_became_peered":"2024-03-12T02:54:03.227429+0000","last_unstale":"2024-03-12T02:54:31.002857+0000","last_undegraded":"2024-03-12T02:54:31.002857+0000","last_fullsized":"2024-03-12T02:54:31.002857+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:07.072163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.001939+0000","last_change":"2024-03-12T02:54:05.237039+0000","last_active":"2024-03-12T02:54:31.001939+0000","last_peered":"2024-03-12T02:54:31.001939+0000","last_clean":"2024-03-12T02:54:31.001939+0000","last_became_active":"2024-03-12T02:54:05.234567+0000","last_became_peered":"2024-03-12T02:54:05.234567+0000","last_unstale":"2024-03-12T02:54:31.001939+0000","last_undegraded":"2024-03-12T02:54:31.001939+0000","last_fullsized":"2024-03-12T02:54:31.001939+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:22:26.866756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005289+0000","last_change":"2024-03-12T02:54:09.263222+0000","last_active":"2024-03-12T02:54:31.005289+0000","last_peered":"2024-03-12T02:54:31.005289+0000","last_clean":"2024-03-12T02:54:31.005289+0000","last_became_active":"2024-03-12T02:54:09.262639+0000","last_became_peered":"2024-03-12T02:54:09.262639+0000","last_unstale":"2024-03-12T02:54:31.005289+0000","last_undegraded":"2024-03-12T02:54:31.005289+0000","last_fullsized":"2024-03-12T02:54:31.005289+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:34:53.666069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004523+0000","last_change":"2024-03-12T02:54:07.259689+0000","last_active":"2024-03-12T02:54:31.004523+0000","last_peered":"2024-03-12T02:54:31.004523+0000","last_clean":"2024-03-12T02:54:31.004523+0000","last_became_active":"2024-03-12T02:54:07.259373+0000","last_became_peered":"2024-03-12T02:54:07.259373+0000","last_unstale":"2024-03-12T02:54:31.004523+0000","last_undegraded":"2024-03-12T02:54:31.004523+0000","last_fullsized":"2024-03-12T02:54:31.004523+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:14:23.466948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006396+0000","last_change":"2024-03-12T02:54:03.222198+0000","last_active":"2024-03-12T02:54:31.006396+0000","last_peered":"2024-03-12T02:54:31.006396+0000","last_clean":"2024-03-12T02:54:31.006396+0000","last_became_active":"2024-03-12T02:54:03.221869+0000","last_became_peered":"2024-03-12T02:54:03.221869+0000","last_unstale":"2024-03-12T02:54:31.006396+0000","last_undegraded":"2024-03-12T02:54:31.006396+0000","last_fullsized":"2024-03-12T02:54:31.006396+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:50:58.355917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'39","reported_seq":96,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002358+0000","last_change":"2024-03-12T02:54:05.232425+0000","last_active":"2024-03-12T02:54:31.002358+0000","last_peered":"2024-03-12T02:54:31.002358+0000","last_clean":"2024-03-12T02:54:31.002358+0000","last_became_active":"2024-03-12T02:54:05.231864+0000","last_became_peered":"2024-03-12T02:54:05.231864+0000","last_unstale":"2024-03-12T02:54:31.002358+0000","last_undegraded":"2024-03-12T02:54:31.002358+0000","last_fullsized":"2024-03-12T02:54:31.002358+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:34:57.218306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004708+0000","last_change":"2024-03-12T02:54:09.264184+0000","last_active":"2024-03-12T02:54:31.004708+0000","last_peered":"2024-03-12T02:54:31.004708+0000","last_clean":"2024-03-12T02:54:31.004708+0000","last_became_active":"2024-03-12T02:54:09.263684+0000","last_became_peered":"2024-03-12T02:54:09.263684+0000","last_unstale":"2024-03-12T02:54:31.004708+0000","last_undegraded":"2024-03-12T02:54:31.004708+0000","last_fullsized":"2024-03-12T02:54:31.004708+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:10:20.734586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.001779+0000","last_change":"2024-03-12T02:54:07.253843+0000","last_active":"2024-03-12T02:54:31.001779+0000","last_peered":"2024-03-12T02:54:31.001779+0000","last_clean":"2024-03-12T02:54:31.001779+0000","last_became_active":"2024-03-12T02:54:07.252732+0000","last_became_peered":"2024-03-12T02:54:07.252732+0000","last_unstale":"2024-03-12T02:54:31.001779+0000","last_undegraded":"2024-03-12T02:54:31.001779+0000","last_fullsized":"2024-03-12T02:54:31.001779+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:35:23.782833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006630+0000","last_change":"2024-03-12T02:54:03.228617+0000","last_active":"2024-03-12T02:54:31.006630+0000","last_peered":"2024-03-12T02:54:31.006630+0000","last_clean":"2024-03-12T02:54:31.006630+0000","last_became_active":"2024-03-12T02:54:03.228302+0000","last_became_peered":"2024-03-12T02:54:03.228302+0000","last_unstale":"2024-03-12T02:54:31.006630+0000","last_undegraded":"2024-03-12T02:54:31.006630+0000","last_fullsized":"2024-03-12T02:54:31.006630+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:35:20.881427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'21","reported_seq":63,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006822+0000","last_change":"2024-03-12T02:54:05.238673+0000","last_active":"2024-03-12T02:54:31.006822+0000","last_peered":"2024-03-12T02:54:31.006822+0000","last_clean":"2024-03-12T02:54:31.006822+0000","last_became_active":"2024-03-12T02:54:05.238341+0000","last_became_peered":"2024-03-12T02:54:05.238341+0000","last_unstale":"2024-03-12T02:54:31.006822+0000","last_undegraded":"2024-03-12T02:54:31.006822+0000","last_fullsized":"2024-03-12T02:54:31.006822+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:32:03.077399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004055+0000","last_change":"2024-03-12T02:54:09.263320+0000","last_active":"2024-03-12T02:54:31.004055+0000","last_peered":"2024-03-12T02:54:31.004055+0000","last_clean":"2024-03-12T02:54:31.004055+0000","last_became_active":"2024-03-12T02:54:09.262528+0000","last_became_peered":"2024-03-12T02:54:09.262528+0000","last_unstale":"2024-03-12T02:54:31.004055+0000","last_undegraded":"2024-03-12T02:54:31.004055+0000","last_fullsized":"2024-03-12T02:54:31.004055+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:29:22.197950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002020+0000","last_change":"2024-03-12T02:54:07.256470+0000","last_active":"2024-03-12T02:54:31.002020+0000","last_peered":"2024-03-12T02:54:31.002020+0000","last_clean":"2024-03-12T02:54:31.002020+0000","last_became_active":"2024-03-12T02:54:07.256135+0000","last_became_peered":"2024-03-12T02:54:07.256135+0000","last_unstale":"2024-03-12T02:54:31.002020+0000","last_undegraded":"2024-03-12T02:54:31.002020+0000","last_fullsized":"2024-03-12T02:54:31.002020+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:34:43.356196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006572+0000","last_change":"2024-03-12T02:54:03.227079+0000","last_active":"2024-03-12T02:54:31.006572+0000","last_peered":"2024-03-12T02:54:31.006572+0000","last_clean":"2024-03-12T02:54:31.006572+0000","last_became_active":"2024-03-12T02:54:03.226635+0000","last_became_peered":"2024-03-12T02:54:03.226635+0000","last_unstale":"2024-03-12T02:54:31.006572+0000","last_undegraded":"2024-03-12T02:54:31.006572+0000","last_fullsized":"2024-03-12T02:54:31.006572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:36:29.069294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'12","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005729+0000","last_change":"2024-03-12T02:54:05.235495+0000","last_active":"2024-03-12T02:54:31.005729+0000","last_peered":"2024-03-12T02:54:31.005729+0000","last_clean":"2024-03-12T02:54:31.005729+0000","last_became_active":"2024-03-12T02:54:05.235162+0000","last_became_peered":"2024-03-12T02:54:05.235162+0000","last_unstale":"2024-03-12T02:54:31.005729+0000","last_undegraded":"2024-03-12T02:54:31.005729+0000","last_fullsized":"2024-03-12T02:54:31.005729+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:26:08.135711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006390+0000","last_change":"2024-03-12T02:54:09.262079+0000","last_active":"2024-03-12T02:54:31.006390+0000","last_peered":"2024-03-12T02:54:31.006390+0000","last_clean":"2024-03-12T02:54:31.006390+0000","last_became_active":"2024-03-12T02:54:09.261319+0000","last_became_peered":"2024-03-12T02:54:09.261319+0000","last_unstale":"2024-03-12T02:54:31.006390+0000","last_undegraded":"2024-03-12T02:54:31.006390+0000","last_fullsized":"2024-03-12T02:54:31.006390+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:59:00.259665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004661+0000","last_change":"2024-03-12T02:54:07.255224+0000","last_active":"2024-03-12T02:54:31.004661+0000","last_peered":"2024-03-12T02:54:31.004661+0000","last_clean":"2024-03-12T02:54:31.004661+0000","last_became_active":"2024-03-12T02:54:07.254656+0000","last_became_peered":"2024-03-12T02:54:07.254656+0000","last_unstale":"2024-03-12T02:54:31.004661+0000","last_undegraded":"2024-03-12T02:54:31.004661+0000","last_fullsized":"2024-03-12T02:54:31.004661+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:51:47.700475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008721+0000","last_change":"2024-03-12T02:54:03.223436+0000","last_active":"2024-03-12T02:54:31.008721+0000","last_peered":"2024-03-12T02:54:31.008721+0000","last_clean":"2024-03-12T02:54:31.008721+0000","last_became_active":"2024-03-12T02:54:03.222345+0000","last_became_peered":"2024-03-12T02:54:03.222345+0000","last_unstale":"2024-03-12T02:54:31.008721+0000","last_undegraded":"2024-03-12T02:54:31.008721+0000","last_fullsized":"2024-03-12T02:54:31.008721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:54:18.905498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'22","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005664+0000","last_change":"2024-03-12T02:54:05.238022+0000","last_active":"2024-03-12T02:54:31.005664+0000","last_peered":"2024-03-12T02:54:31.005664+0000","last_clean":"2024-03-12T02:54:31.005664+0000","last_became_active":"2024-03-12T02:54:05.237356+0000","last_became_peered":"2024-03-12T02:54:05.237356+0000","last_unstale":"2024-03-12T02:54:31.005664+0000","last_undegraded":"2024-03-12T02:54:31.005664+0000","last_fullsized":"2024-03-12T02:54:31.005664+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:23:57.767122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006609+0000","last_change":"2024-03-12T02:54:09.261304+0000","last_active":"2024-03-12T02:54:31.006609+0000","last_peered":"2024-03-12T02:54:31.006609+0000","last_clean":"2024-03-12T02:54:31.006609+0000","last_became_active":"2024-03-12T02:54:09.261004+0000","last_became_peered":"2024-03-12T02:54:09.261004+0000","last_unstale":"2024-03-12T02:54:31.006609+0000","last_undegraded":"2024-03-12T02:54:31.006609+0000","last_fullsized":"2024-03-12T02:54:31.006609+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:17:36.305661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002158+0000","last_change":"2024-03-12T02:54:07.254115+0000","last_active":"2024-03-12T02:54:31.002158+0000","last_peered":"2024-03-12T02:54:31.002158+0000","last_clean":"2024-03-12T02:54:31.002158+0000","last_became_active":"2024-03-12T02:54:07.253729+0000","last_became_peered":"2024-03-12T02:54:07.253729+0000","last_unstale":"2024-03-12T02:54:31.002158+0000","last_undegraded":"2024-03-12T02:54:31.002158+0000","last_fullsized":"2024-03-12T02:54:31.002158+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:44:09.014023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005306+0000","last_change":"2024-03-12T02:54:03.227350+0000","last_active":"2024-03-12T02:54:31.005306+0000","last_peered":"2024-03-12T02:54:31.005306+0000","last_clean":"2024-03-12T02:54:31.005306+0000","last_became_active":"2024-03-12T02:54:03.226910+0000","last_became_peered":"2024-03-12T02:54:03.226910+0000","last_unstale":"2024-03-12T02:54:31.005306+0000","last_undegraded":"2024-03-12T02:54:31.005306+0000","last_fullsized":"2024-03-12T02:54:31.005306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:10:21.918099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008279+0000","last_change":"2024-03-12T02:54:05.234490+0000","last_active":"2024-03-12T02:54:31.008279+0000","last_peered":"2024-03-12T02:54:31.008279+0000","last_clean":"2024-03-12T02:54:31.008279+0000","last_became_active":"2024-03-12T02:54:05.234122+0000","last_became_peered":"2024-03-12T02:54:05.234122+0000","last_unstale":"2024-03-12T02:54:31.008279+0000","last_undegraded":"2024-03-12T02:54:31.008279+0000","last_fullsized":"2024-03-12T02:54:31.008279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:10:20.714956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003405+0000","last_change":"2024-03-12T02:54:09.258361+0000","last_active":"2024-03-12T02:54:31.003405+0000","last_peered":"2024-03-12T02:54:31.003405+0000","last_clean":"2024-03-12T02:54:31.003405+0000","last_became_active":"2024-03-12T02:54:09.257929+0000","last_became_peered":"2024-03-12T02:54:09.257929+0000","last_unstale":"2024-03-12T02:54:31.003405+0000","last_undegraded":"2024-03-12T02:54:31.003405+0000","last_fullsized":"2024-03-12T02:54:31.003405+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:14:21.972211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.268620+0000","last_change":"2024-03-12T02:54:07.252459+0000","last_active":"2024-03-12T02:54:33.268620+0000","last_peered":"2024-03-12T02:54:33.268620+0000","last_clean":"2024-03-12T02:54:33.268620+0000","last_became_active":"2024-03-12T02:54:07.252184+0000","last_became_peered":"2024-03-12T02:54:07.252184+0000","last_unstale":"2024-03-12T02:54:33.268620+0000","last_undegraded":"2024-03-12T02:54:33.268620+0000","last_fullsized":"2024-03-12T02:54:33.268620+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:29.703459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007045+0000","last_change":"2024-03-12T02:54:03.223425+0000","last_active":"2024-03-12T02:54:31.007045+0000","last_peered":"2024-03-12T02:54:31.007045+0000","last_clean":"2024-03-12T02:54:31.007045+0000","last_became_active":"2024-03-12T02:54:03.222484+0000","last_became_peered":"2024-03-12T02:54:03.222484+0000","last_unstale":"2024-03-12T02:54:31.007045+0000","last_undegraded":"2024-03-12T02:54:31.007045+0000","last_fullsized":"2024-03-12T02:54:31.007045+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:32:32.455804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002503+0000","last_change":"2024-03-12T02:54:05.232774+0000","last_active":"2024-03-12T02:54:31.002503+0000","last_peered":"2024-03-12T02:54:31.002503+0000","last_clean":"2024-03-12T02:54:31.002503+0000","last_became_active":"2024-03-12T02:54:05.232449+0000","last_became_peered":"2024-03-12T02:54:05.232449+0000","last_unstale":"2024-03-12T02:54:31.002503+0000","last_undegraded":"2024-03-12T02:54:31.002503+0000","last_fullsized":"2024-03-12T02:54:31.002503+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:03:08.187506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005588+0000","last_change":"2024-03-12T02:54:09.264192+0000","last_active":"2024-03-12T02:54:31.005588+0000","last_peered":"2024-03-12T02:54:31.005588+0000","last_clean":"2024-03-12T02:54:31.005588+0000","last_became_active":"2024-03-12T02:54:09.263557+0000","last_became_peered":"2024-03-12T02:54:09.263557+0000","last_unstale":"2024-03-12T02:54:31.005588+0000","last_undegraded":"2024-03-12T02:54:31.005588+0000","last_fullsized":"2024-03-12T02:54:31.005588+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:27.630121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.268576+0000","last_change":"2024-03-12T02:54:07.255094+0000","last_active":"2024-03-12T02:54:33.268576+0000","last_peered":"2024-03-12T02:54:33.268576+0000","last_clean":"2024-03-12T02:54:33.268576+0000","last_became_active":"2024-03-12T02:54:07.254776+0000","last_became_peered":"2024-03-12T02:54:07.254776+0000","last_unstale":"2024-03-12T02:54:33.268576+0000","last_undegraded":"2024-03-12T02:54:33.268576+0000","last_fullsized":"2024-03-12T02:54:33.268576+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:34:04.511468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006601+0000","last_change":"2024-03-12T02:54:03.225046+0000","last_active":"2024-03-12T02:54:31.006601+0000","last_peered":"2024-03-12T02:54:31.006601+0000","last_clean":"2024-03-12T02:54:31.006601+0000","last_became_active":"2024-03-12T02:54:03.223354+0000","last_became_peered":"2024-03-12T02:54:03.223354+0000","last_unstale":"2024-03-12T02:54:31.006601+0000","last_undegraded":"2024-03-12T02:54:31.006601+0000","last_fullsized":"2024-03-12T02:54:31.006601+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:30:33.792780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004911+0000","last_change":"2024-03-12T02:54:05.236021+0000","last_active":"2024-03-12T02:54:31.004911+0000","last_peered":"2024-03-12T02:54:31.004911+0000","last_clean":"2024-03-12T02:54:31.004911+0000","last_became_active":"2024-03-12T02:54:05.235517+0000","last_became_peered":"2024-03-12T02:54:05.235517+0000","last_unstale":"2024-03-12T02:54:31.004911+0000","last_undegraded":"2024-03-12T02:54:31.004911+0000","last_fullsized":"2024-03-12T02:54:31.004911+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:46.468603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005907+0000","last_change":"2024-03-12T02:54:09.257663+0000","last_active":"2024-03-12T02:54:31.005907+0000","last_peered":"2024-03-12T02:54:31.005907+0000","last_clean":"2024-03-12T02:54:31.005907+0000","last_became_active":"2024-03-12T02:54:09.257321+0000","last_became_peered":"2024-03-12T02:54:09.257321+0000","last_unstale":"2024-03-12T02:54:31.005907+0000","last_undegraded":"2024-03-12T02:54:31.005907+0000","last_fullsized":"2024-03-12T02:54:31.005907+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:19:25.103143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005489+0000","last_change":"2024-03-12T02:54:07.259511+0000","last_active":"2024-03-12T02:54:31.005489+0000","last_peered":"2024-03-12T02:54:31.005489+0000","last_clean":"2024-03-12T02:54:31.005489+0000","last_became_active":"2024-03-12T02:54:07.259200+0000","last_became_peered":"2024-03-12T02:54:07.259200+0000","last_unstale":"2024-03-12T02:54:31.005489+0000","last_undegraded":"2024-03-12T02:54:31.005489+0000","last_fullsized":"2024-03-12T02:54:31.005489+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:23:48.027202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003989+0000","last_change":"2024-03-12T02:54:03.221775+0000","last_active":"2024-03-12T02:54:31.003989+0000","last_peered":"2024-03-12T02:54:31.003989+0000","last_clean":"2024-03-12T02:54:31.003989+0000","last_became_active":"2024-03-12T02:54:03.220660+0000","last_became_peered":"2024-03-12T02:54:03.220660+0000","last_unstale":"2024-03-12T02:54:31.003989+0000","last_undegraded":"2024-03-12T02:54:31.003989+0000","last_fullsized":"2024-03-12T02:54:31.003989+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:02:07.511766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004122+0000","last_change":"2024-03-12T02:54:05.238734+0000","last_active":"2024-03-12T02:54:31.004122+0000","last_peered":"2024-03-12T02:54:31.004122+0000","last_clean":"2024-03-12T02:54:31.004122+0000","last_became_active":"2024-03-12T02:54:05.238417+0000","last_became_peered":"2024-03-12T02:54:05.238417+0000","last_unstale":"2024-03-12T02:54:31.004122+0000","last_undegraded":"2024-03-12T02:54:31.004122+0000","last_fullsized":"2024-03-12T02:54:31.004122+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:23:18.321847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005231+0000","last_change":"2024-03-12T02:54:09.261271+0000","last_active":"2024-03-12T02:54:31.005231+0000","last_peered":"2024-03-12T02:54:31.005231+0000","last_clean":"2024-03-12T02:54:31.005231+0000","last_became_active":"2024-03-12T02:54:09.260913+0000","last_became_peered":"2024-03-12T02:54:09.260913+0000","last_unstale":"2024-03-12T02:54:31.005231+0000","last_undegraded":"2024-03-12T02:54:31.005231+0000","last_fullsized":"2024-03-12T02:54:31.005231+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:50:53.351133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004610+0000","last_change":"2024-03-12T02:54:07.259642+0000","last_active":"2024-03-12T02:54:31.004610+0000","last_peered":"2024-03-12T02:54:31.004610+0000","last_clean":"2024-03-12T02:54:31.004610+0000","last_became_active":"2024-03-12T02:54:07.259240+0000","last_became_peered":"2024-03-12T02:54:07.259240+0000","last_unstale":"2024-03-12T02:54:31.004610+0000","last_undegraded":"2024-03-12T02:54:31.004610+0000","last_fullsized":"2024-03-12T02:54:31.004610+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:55:16.464794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006498+0000","last_change":"2024-03-12T02:54:03.223127+0000","last_active":"2024-03-12T02:54:31.006498+0000","last_peered":"2024-03-12T02:54:31.006498+0000","last_clean":"2024-03-12T02:54:31.006498+0000","last_became_active":"2024-03-12T02:54:03.222068+0000","last_became_peered":"2024-03-12T02:54:03.222068+0000","last_unstale":"2024-03-12T02:54:31.006498+0000","last_undegraded":"2024-03-12T02:54:31.006498+0000","last_fullsized":"2024-03-12T02:54:31.006498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:59:01.816434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007235+0000","last_change":"2024-03-12T02:54:05.236063+0000","last_active":"2024-03-12T02:54:31.007235+0000","last_peered":"2024-03-12T02:54:31.007235+0000","last_clean":"2024-03-12T02:54:31.007235+0000","last_became_active":"2024-03-12T02:54:05.235767+0000","last_became_peered":"2024-03-12T02:54:05.235767+0000","last_unstale":"2024-03-12T02:54:31.007235+0000","last_undegraded":"2024-03-12T02:54:31.007235+0000","last_fullsized":"2024-03-12T02:54:31.007235+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:35:34.341194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007302+0000","last_change":"2024-03-12T02:54:09.261429+0000","last_active":"2024-03-12T02:54:31.007302+0000","last_peered":"2024-03-12T02:54:31.007302+0000","last_clean":"2024-03-12T02:54:31.007302+0000","last_became_active":"2024-03-12T02:54:09.261104+0000","last_became_peered":"2024-03-12T02:54:09.261104+0000","last_unstale":"2024-03-12T02:54:31.007302+0000","last_undegraded":"2024-03-12T02:54:31.007302+0000","last_fullsized":"2024-03-12T02:54:31.007302+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:23:04.807606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008651+0000","last_change":"2024-03-12T02:54:03.225515+0000","last_active":"2024-03-12T02:54:31.008651+0000","last_peered":"2024-03-12T02:54:31.008651+0000","last_clean":"2024-03-12T02:54:31.008651+0000","last_became_active":"2024-03-12T02:54:03.223339+0000","last_became_peered":"2024-03-12T02:54:03.223339+0000","last_unstale":"2024-03-12T02:54:31.008651+0000","last_undegraded":"2024-03-12T02:54:31.008651+0000","last_fullsized":"2024-03-12T02:54:31.008651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:25.270252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008711+0000","last_change":"2024-03-12T02:54:05.236067+0000","last_active":"2024-03-12T02:54:31.008711+0000","last_peered":"2024-03-12T02:54:31.008711+0000","last_clean":"2024-03-12T02:54:31.008711+0000","last_became_active":"2024-03-12T02:54:05.235692+0000","last_became_peered":"2024-03-12T02:54:05.235692+0000","last_unstale":"2024-03-12T02:54:31.008711+0000","last_undegraded":"2024-03-12T02:54:31.008711+0000","last_fullsized":"2024-03-12T02:54:31.008711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:50.573680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.268636+0000","last_change":"2024-03-12T02:54:07.255405+0000","last_active":"2024-03-12T02:54:33.268636+0000","last_peered":"2024-03-12T02:54:33.268636+0000","last_clean":"2024-03-12T02:54:33.268636+0000","last_became_active":"2024-03-12T02:54:07.255072+0000","last_became_peered":"2024-03-12T02:54:07.255072+0000","last_unstale":"2024-03-12T02:54:33.268636+0000","last_undegraded":"2024-03-12T02:54:33.268636+0000","last_fullsized":"2024-03-12T02:54:33.268636+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:50:43.443544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002347+0000","last_change":"2024-03-12T02:54:09.258261+0000","last_active":"2024-03-12T02:54:31.002347+0000","last_peered":"2024-03-12T02:54:31.002347+0000","last_clean":"2024-03-12T02:54:31.002347+0000","last_became_active":"2024-03-12T02:54:09.257861+0000","last_became_peered":"2024-03-12T02:54:09.257861+0000","last_unstale":"2024-03-12T02:54:31.002347+0000","last_undegraded":"2024-03-12T02:54:31.002347+0000","last_fullsized":"2024-03-12T02:54:31.002347+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:32:05.076028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004134+0000","last_change":"2024-03-12T02:54:03.228357+0000","last_active":"2024-03-12T02:54:31.004134+0000","last_peered":"2024-03-12T02:54:31.004134+0000","last_clean":"2024-03-12T02:54:31.004134+0000","last_became_active":"2024-03-12T02:54:03.228003+0000","last_became_peered":"2024-03-12T02:54:03.228003+0000","last_unstale":"2024-03-12T02:54:31.004134+0000","last_undegraded":"2024-03-12T02:54:31.004134+0000","last_fullsized":"2024-03-12T02:54:31.004134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:42:28.054205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":53,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.275472+0000","last_change":"2024-03-12T02:54:05.236101+0000","last_active":"2024-03-12T02:54:33.275472+0000","last_peered":"2024-03-12T02:54:33.275472+0000","last_clean":"2024-03-12T02:54:33.275472+0000","last_became_active":"2024-03-12T02:54:05.235549+0000","last_became_peered":"2024-03-12T02:54:05.235549+0000","last_unstale":"2024-03-12T02:54:33.275472+0000","last_undegraded":"2024-03-12T02:54:33.275472+0000","last_fullsized":"2024-03-12T02:54:33.275472+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:27:39.230654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006419+0000","last_change":"2024-03-12T02:54:07.251833+0000","last_active":"2024-03-12T02:54:31.006419+0000","last_peered":"2024-03-12T02:54:31.006419+0000","last_clean":"2024-03-12T02:54:31.006419+0000","last_became_active":"2024-03-12T02:54:07.250566+0000","last_became_peered":"2024-03-12T02:54:07.250566+0000","last_unstale":"2024-03-12T02:54:31.006419+0000","last_undegraded":"2024-03-12T02:54:31.006419+0000","last_fullsized":"2024-03-12T02:54:31.006419+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:01:20.257983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004338+0000","last_change":"2024-03-12T02:54:09.258514+0000","last_active":"2024-03-12T02:54:31.004338+0000","last_peered":"2024-03-12T02:54:31.004338+0000","last_clean":"2024-03-12T02:54:31.004338+0000","last_became_active":"2024-03-12T02:54:09.258199+0000","last_became_peered":"2024-03-12T02:54:09.258199+0000","last_unstale":"2024-03-12T02:54:31.004338+0000","last_undegraded":"2024-03-12T02:54:31.004338+0000","last_fullsized":"2024-03-12T02:54:31.004338+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:51:09.391155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004421+0000","last_change":"2024-03-12T02:54:03.222082+0000","last_active":"2024-03-12T02:54:31.004421+0000","last_peered":"2024-03-12T02:54:31.004421+0000","last_clean":"2024-03-12T02:54:31.004421+0000","last_became_active":"2024-03-12T02:54:03.221696+0000","last_became_peered":"2024-03-12T02:54:03.221696+0000","last_unstale":"2024-03-12T02:54:31.004421+0000","last_undegraded":"2024-03-12T02:54:31.004421+0000","last_fullsized":"2024-03-12T02:54:31.004421+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:17:18.708749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'25","reported_seq":71,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005136+0000","last_change":"2024-03-12T02:54:05.235363+0000","last_active":"2024-03-12T02:54:31.005136+0000","last_peered":"2024-03-12T02:54:31.005136+0000","last_clean":"2024-03-12T02:54:31.005136+0000","last_became_active":"2024-03-12T02:54:05.234994+0000","last_became_peered":"2024-03-12T02:54:05.234994+0000","last_unstale":"2024-03-12T02:54:31.005136+0000","last_undegraded":"2024-03-12T02:54:31.005136+0000","last_fullsized":"2024-03-12T02:54:31.005136+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:33:56.680419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003492+0000","last_change":"2024-03-12T02:54:07.256841+0000","last_active":"2024-03-12T02:54:31.003492+0000","last_peered":"2024-03-12T02:54:31.003492+0000","last_clean":"2024-03-12T02:54:31.003492+0000","last_became_active":"2024-03-12T02:54:07.256485+0000","last_became_peered":"2024-03-12T02:54:07.256485+0000","last_unstale":"2024-03-12T02:54:31.003492+0000","last_undegraded":"2024-03-12T02:54:31.003492+0000","last_fullsized":"2024-03-12T02:54:31.003492+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:52:10.673498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005395+0000","last_change":"2024-03-12T02:54:09.261808+0000","last_active":"2024-03-12T02:54:31.005395+0000","last_peered":"2024-03-12T02:54:31.005395+0000","last_clean":"2024-03-12T02:54:31.005395+0000","last_became_active":"2024-03-12T02:54:09.261452+0000","last_became_peered":"2024-03-12T02:54:09.261452+0000","last_unstale":"2024-03-12T02:54:31.005395+0000","last_undegraded":"2024-03-12T02:54:31.005395+0000","last_fullsized":"2024-03-12T02:54:31.005395+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:38:35.593312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'30","reported_seq":79,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006114+0000","last_change":"2024-03-12T02:54:05.234492+0000","last_active":"2024-03-12T02:54:31.006114+0000","last_peered":"2024-03-12T02:54:31.006114+0000","last_clean":"2024-03-12T02:54:31.006114+0000","last_became_active":"2024-03-12T02:54:05.234157+0000","last_became_peered":"2024-03-12T02:54:05.234157+0000","last_unstale":"2024-03-12T02:54:31.006114+0000","last_undegraded":"2024-03-12T02:54:31.006114+0000","last_fullsized":"2024-03-12T02:54:31.006114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:32:27.274904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002541+0000","last_change":"2024-03-12T02:54:03.227096+0000","last_active":"2024-03-12T02:54:31.002541+0000","last_peered":"2024-03-12T02:54:31.002541+0000","last_clean":"2024-03-12T02:54:31.002541+0000","last_became_active":"2024-03-12T02:54:03.226790+0000","last_became_peered":"2024-03-12T02:54:03.226790+0000","last_unstale":"2024-03-12T02:54:31.002541+0000","last_undegraded":"2024-03-12T02:54:31.002541+0000","last_fullsized":"2024-03-12T02:54:31.002541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:26:49.222298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005696+0000","last_change":"2024-03-12T02:54:07.259609+0000","last_active":"2024-03-12T02:54:31.005696+0000","last_peered":"2024-03-12T02:54:31.005696+0000","last_clean":"2024-03-12T02:54:31.005696+0000","last_became_active":"2024-03-12T02:54:07.259233+0000","last_became_peered":"2024-03-12T02:54:07.259233+0000","last_unstale":"2024-03-12T02:54:31.005696+0000","last_undegraded":"2024-03-12T02:54:31.005696+0000","last_fullsized":"2024-03-12T02:54:31.005696+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:31:04.355212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007075+0000","last_change":"2024-03-12T02:54:09.261527+0000","last_active":"2024-03-12T02:54:31.007075+0000","last_peered":"2024-03-12T02:54:31.007075+0000","last_clean":"2024-03-12T02:54:31.007075+0000","last_became_active":"2024-03-12T02:54:09.261128+0000","last_became_peered":"2024-03-12T02:54:09.261128+0000","last_unstale":"2024-03-12T02:54:31.007075+0000","last_undegraded":"2024-03-12T02:54:31.007075+0000","last_fullsized":"2024-03-12T02:54:31.007075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:50.566784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'16","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002632+0000","last_change":"2024-03-12T02:54:05.232529+0000","last_active":"2024-03-12T02:54:31.002632+0000","last_peered":"2024-03-12T02:54:31.002632+0000","last_clean":"2024-03-12T02:54:31.002632+0000","last_became_active":"2024-03-12T02:54:05.231838+0000","last_became_peered":"2024-03-12T02:54:05.231838+0000","last_unstale":"2024-03-12T02:54:31.002632+0000","last_undegraded":"2024-03-12T02:54:31.002632+0000","last_fullsized":"2024-03-12T02:54:31.002632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:25:51.846591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004903+0000","last_change":"2024-03-12T02:54:03.222035+0000","last_active":"2024-03-12T02:54:31.004903+0000","last_peered":"2024-03-12T02:54:31.004903+0000","last_clean":"2024-03-12T02:54:31.004903+0000","last_became_active":"2024-03-12T02:54:03.221065+0000","last_became_peered":"2024-03-12T02:54:03.221065+0000","last_unstale":"2024-03-12T02:54:31.004903+0000","last_undegraded":"2024-03-12T02:54:31.004903+0000","last_fullsized":"2024-03-12T02:54:31.004903+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:22:15.530498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.109182+0000","last_change":"2024-03-12T02:53:52.239954+0000","last_active":"2024-03-12T02:54:33.109182+0000","last_peered":"2024-03-12T02:54:33.109182+0000","last_clean":"2024-03-12T02:54:33.109182+0000","last_became_active":"2024-03-12T02:53:52.239662+0000","last_became_peered":"2024-03-12T02:53:52.239662+0000","last_unstale":"2024-03-12T02:54:33.109182+0000","last_undegraded":"2024-03-12T02:54:33.109182+0000","last_fullsized":"2024-03-12T02:54:33.109182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:53:51.228013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:53:51.228013+0000","last_clean_scrub_stamp":"2024-03-12T02:53:51.228013+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:40.204759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006379+0000","last_change":"2024-03-12T02:54:07.255847+0000","last_active":"2024-03-12T02:54:31.006379+0000","last_peered":"2024-03-12T02:54:31.006379+0000","last_clean":"2024-03-12T02:54:31.006379+0000","last_became_active":"2024-03-12T02:54:07.255500+0000","last_became_peered":"2024-03-12T02:54:07.255500+0000","last_unstale":"2024-03-12T02:54:31.006379+0000","last_undegraded":"2024-03-12T02:54:31.006379+0000","last_fullsized":"2024-03-12T02:54:31.006379+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:20.715553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005388+0000","last_change":"2024-03-12T02:54:09.257867+0000","last_active":"2024-03-12T02:54:31.005388+0000","last_peered":"2024-03-12T02:54:31.005388+0000","last_clean":"2024-03-12T02:54:31.005388+0000","last_became_active":"2024-03-12T02:54:09.257396+0000","last_became_peered":"2024-03-12T02:54:09.257396+0000","last_unstale":"2024-03-12T02:54:31.005388+0000","last_undegraded":"2024-03-12T02:54:31.005388+0000","last_fullsized":"2024-03-12T02:54:31.005388+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:20.260521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'20","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002574+0000","last_change":"2024-03-12T02:54:05.232455+0000","last_active":"2024-03-12T02:54:31.002574+0000","last_peered":"2024-03-12T02:54:31.002574+0000","last_clean":"2024-03-12T02:54:31.002574+0000","last_became_active":"2024-03-12T02:54:05.231898+0000","last_became_peered":"2024-03-12T02:54:05.231898+0000","last_unstale":"2024-03-12T02:54:31.002574+0000","last_undegraded":"2024-03-12T02:54:31.002574+0000","last_fullsized":"2024-03-12T02:54:31.002574+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:35:55.130489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003980+0000","last_change":"2024-03-12T02:54:03.227441+0000","last_active":"2024-03-12T02:54:31.003980+0000","last_peered":"2024-03-12T02:54:31.003980+0000","last_clean":"2024-03-12T02:54:31.003980+0000","last_became_active":"2024-03-12T02:54:03.227072+0000","last_became_peered":"2024-03-12T02:54:03.227072+0000","last_unstale":"2024-03-12T02:54:31.003980+0000","last_undegraded":"2024-03-12T02:54:31.003980+0000","last_fullsized":"2024-03-12T02:54:31.003980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:00:04.018362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008555+0000","last_change":"2024-03-12T02:54:07.256791+0000","last_active":"2024-03-12T02:54:31.008555+0000","last_peered":"2024-03-12T02:54:31.008555+0000","last_clean":"2024-03-12T02:54:31.008555+0000","last_became_active":"2024-03-12T02:54:07.256510+0000","last_became_peered":"2024-03-12T02:54:07.256510+0000","last_unstale":"2024-03-12T02:54:31.008555+0000","last_undegraded":"2024-03-12T02:54:31.008555+0000","last_fullsized":"2024-03-12T02:54:31.008555+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:59:13.677601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004256+0000","last_change":"2024-03-12T02:54:09.256439+0000","last_active":"2024-03-12T02:54:31.004256+0000","last_peered":"2024-03-12T02:54:31.004256+0000","last_clean":"2024-03-12T02:54:31.004256+0000","last_became_active":"2024-03-12T02:54:09.256076+0000","last_became_peered":"2024-03-12T02:54:09.256076+0000","last_unstale":"2024-03-12T02:54:31.004256+0000","last_undegraded":"2024-03-12T02:54:31.004256+0000","last_fullsized":"2024-03-12T02:54:31.004256+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:56:13.116283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008416+0000","last_change":"2024-03-12T02:54:03.223170+0000","last_active":"2024-03-12T02:54:31.008416+0000","last_peered":"2024-03-12T02:54:31.008416+0000","last_clean":"2024-03-12T02:54:31.008416+0000","last_became_active":"2024-03-12T02:54:03.222205+0000","last_became_peered":"2024-03-12T02:54:03.222205+0000","last_unstale":"2024-03-12T02:54:31.008416+0000","last_undegraded":"2024-03-12T02:54:31.008416+0000","last_fullsized":"2024-03-12T02:54:31.008416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:58.308197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'34","reported_seq":91,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.268464+0000","last_change":"2024-03-12T02:54:05.233098+0000","last_active":"2024-03-12T02:54:33.268464+0000","last_peered":"2024-03-12T02:54:33.268464+0000","last_clean":"2024-03-12T02:54:33.268464+0000","last_became_active":"2024-03-12T02:54:05.232761+0000","last_became_peered":"2024-03-12T02:54:05.232761+0000","last_unstale":"2024-03-12T02:54:33.268464+0000","last_undegraded":"2024-03-12T02:54:33.268464+0000","last_fullsized":"2024-03-12T02:54:33.268464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:21:27.724795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.268597+0000","last_change":"2024-03-12T02:54:07.255825+0000","last_active":"2024-03-12T02:54:33.268597+0000","last_peered":"2024-03-12T02:54:33.268597+0000","last_clean":"2024-03-12T02:54:33.268597+0000","last_became_active":"2024-03-12T02:54:07.255519+0000","last_became_peered":"2024-03-12T02:54:07.255519+0000","last_unstale":"2024-03-12T02:54:33.268597+0000","last_undegraded":"2024-03-12T02:54:33.268597+0000","last_fullsized":"2024-03-12T02:54:33.268597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:25:25.438343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003647+0000","last_change":"2024-03-12T02:54:09.263744+0000","last_active":"2024-03-12T02:54:31.003647+0000","last_peered":"2024-03-12T02:54:31.003647+0000","last_clean":"2024-03-12T02:54:31.003647+0000","last_became_active":"2024-03-12T02:54:09.263328+0000","last_became_peered":"2024-03-12T02:54:09.263328+0000","last_unstale":"2024-03-12T02:54:31.003647+0000","last_undegraded":"2024-03-12T02:54:31.003647+0000","last_fullsized":"2024-03-12T02:54:31.003647+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:35:28.529623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004927+0000","last_change":"2024-03-12T02:54:03.227074+0000","last_active":"2024-03-12T02:54:31.004927+0000","last_peered":"2024-03-12T02:54:31.004927+0000","last_clean":"2024-03-12T02:54:31.004927+0000","last_became_active":"2024-03-12T02:54:03.226718+0000","last_became_peered":"2024-03-12T02:54:03.226718+0000","last_unstale":"2024-03-12T02:54:31.004927+0000","last_undegraded":"2024-03-12T02:54:31.004927+0000","last_fullsized":"2024-03-12T02:54:31.004927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:29:39.215629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'19","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005937+0000","last_change":"2024-03-12T02:54:05.235241+0000","last_active":"2024-03-12T02:54:31.005937+0000","last_peered":"2024-03-12T02:54:31.005937+0000","last_clean":"2024-03-12T02:54:31.005937+0000","last_became_active":"2024-03-12T02:54:05.234850+0000","last_became_peered":"2024-03-12T02:54:05.234850+0000","last_unstale":"2024-03-12T02:54:31.005937+0000","last_undegraded":"2024-03-12T02:54:31.005937+0000","last_fullsized":"2024-03-12T02:54:31.005937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:08:33.940312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.268472+0000","last_change":"2024-03-12T02:54:07.257345+0000","last_active":"2024-03-12T02:54:33.268472+0000","last_peered":"2024-03-12T02:54:33.268472+0000","last_clean":"2024-03-12T02:54:33.268472+0000","last_became_active":"2024-03-12T02:54:07.256929+0000","last_became_peered":"2024-03-12T02:54:07.256929+0000","last_unstale":"2024-03-12T02:54:33.268472+0000","last_undegraded":"2024-03-12T02:54:33.268472+0000","last_fullsized":"2024-03-12T02:54:33.268472+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:17.720166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006987+0000","last_change":"2024-03-12T02:54:09.262431+0000","last_active":"2024-03-12T02:54:31.006987+0000","last_peered":"2024-03-12T02:54:31.006987+0000","last_clean":"2024-03-12T02:54:31.006987+0000","last_became_active":"2024-03-12T02:54:09.262197+0000","last_became_peered":"2024-03-12T02:54:09.262197+0000","last_unstale":"2024-03-12T02:54:31.006987+0000","last_undegraded":"2024-03-12T02:54:31.006987+0000","last_fullsized":"2024-03-12T02:54:31.006987+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:10:17.271515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006330+0000","last_change":"2024-03-12T02:54:03.228518+0000","last_active":"2024-03-12T02:54:31.006330+0000","last_peered":"2024-03-12T02:54:31.006330+0000","last_clean":"2024-03-12T02:54:31.006330+0000","last_became_active":"2024-03-12T02:54:03.228190+0000","last_became_peered":"2024-03-12T02:54:03.228190+0000","last_unstale":"2024-03-12T02:54:31.006330+0000","last_undegraded":"2024-03-12T02:54:31.006330+0000","last_fullsized":"2024-03-12T02:54:31.006330+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:32.314632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005154+0000","last_change":"2024-03-12T02:54:05.235485+0000","last_active":"2024-03-12T02:54:31.005154+0000","last_peered":"2024-03-12T02:54:31.005154+0000","last_clean":"2024-03-12T02:54:31.005154+0000","last_became_active":"2024-03-12T02:54:05.233642+0000","last_became_peered":"2024-03-12T02:54:05.233642+0000","last_unstale":"2024-03-12T02:54:31.005154+0000","last_undegraded":"2024-03-12T02:54:31.005154+0000","last_fullsized":"2024-03-12T02:54:31.005154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:17:25.837049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006306+0000","last_change":"2024-03-12T02:54:07.251600+0000","last_active":"2024-03-12T02:54:31.006306+0000","last_peered":"2024-03-12T02:54:31.006306+0000","last_clean":"2024-03-12T02:54:31.006306+0000","last_became_active":"2024-03-12T02:54:07.250757+0000","last_became_peered":"2024-03-12T02:54:07.250757+0000","last_unstale":"2024-03-12T02:54:31.006306+0000","last_undegraded":"2024-03-12T02:54:31.006306+0000","last_fullsized":"2024-03-12T02:54:31.006306+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:37:40.370418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005541+0000","last_change":"2024-03-12T02:54:09.257788+0000","last_active":"2024-03-12T02:54:31.005541+0000","last_peered":"2024-03-12T02:54:31.005541+0000","last_clean":"2024-03-12T02:54:31.005541+0000","last_became_active":"2024-03-12T02:54:09.257384+0000","last_became_peered":"2024-03-12T02:54:09.257384+0000","last_unstale":"2024-03-12T02:54:31.005541+0000","last_undegraded":"2024-03-12T02:54:31.005541+0000","last_fullsized":"2024-03-12T02:54:31.005541+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:39:51.047538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.275410+0000","last_change":"2024-03-12T02:54:07.256897+0000","last_active":"2024-03-12T02:54:33.275410+0000","last_peered":"2024-03-12T02:54:33.275410+0000","last_clean":"2024-03-12T02:54:33.275410+0000","last_became_active":"2024-03-12T02:54:07.256546+0000","last_became_peered":"2024-03-12T02:54:07.256546+0000","last_unstale":"2024-03-12T02:54:33.275410+0000","last_undegraded":"2024-03-12T02:54:33.275410+0000","last_fullsized":"2024-03-12T02:54:33.275410+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:07.972485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003775+0000","last_change":"2024-03-12T02:54:03.222122+0000","last_active":"2024-03-12T02:54:31.003775+0000","last_peered":"2024-03-12T02:54:31.003775+0000","last_clean":"2024-03-12T02:54:31.003775+0000","last_became_active":"2024-03-12T02:54:03.220839+0000","last_became_peered":"2024-03-12T02:54:03.220839+0000","last_unstale":"2024-03-12T02:54:31.003775+0000","last_undegraded":"2024-03-12T02:54:31.003775+0000","last_fullsized":"2024-03-12T02:54:31.003775+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:25.561859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'24","reported_seq":70,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006813+0000","last_change":"2024-03-12T02:54:05.234527+0000","last_active":"2024-03-12T02:54:31.006813+0000","last_peered":"2024-03-12T02:54:31.006813+0000","last_clean":"2024-03-12T02:54:31.006813+0000","last_became_active":"2024-03-12T02:54:05.234122+0000","last_became_peered":"2024-03-12T02:54:05.234122+0000","last_unstale":"2024-03-12T02:54:31.006813+0000","last_undegraded":"2024-03-12T02:54:31.006813+0000","last_fullsized":"2024-03-12T02:54:31.006813+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:31:33.618716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.009586+0000","last_change":"2024-03-12T02:54:09.265175+0000","last_active":"2024-03-12T02:54:31.009586+0000","last_peered":"2024-03-12T02:54:31.009586+0000","last_clean":"2024-03-12T02:54:31.009586+0000","last_became_active":"2024-03-12T02:54:09.264823+0000","last_became_peered":"2024-03-12T02:54:09.264823+0000","last_unstale":"2024-03-12T02:54:31.009586+0000","last_undegraded":"2024-03-12T02:54:31.009586+0000","last_fullsized":"2024-03-12T02:54:31.009586+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:47:36.980631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005562+0000","last_change":"2024-03-12T02:54:07.256584+0000","last_active":"2024-03-12T02:54:31.005562+0000","last_peered":"2024-03-12T02:54:31.005562+0000","last_clean":"2024-03-12T02:54:31.005562+0000","last_became_active":"2024-03-12T02:54:07.256147+0000","last_became_peered":"2024-03-12T02:54:07.256147+0000","last_unstale":"2024-03-12T02:54:31.005562+0000","last_undegraded":"2024-03-12T02:54:31.005562+0000","last_fullsized":"2024-03-12T02:54:31.005562+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:22:00.388166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002717+0000","last_change":"2024-03-12T02:54:03.227808+0000","last_active":"2024-03-12T02:54:31.002717+0000","last_peered":"2024-03-12T02:54:31.002717+0000","last_clean":"2024-03-12T02:54:31.002717+0000","last_became_active":"2024-03-12T02:54:03.227515+0000","last_became_peered":"2024-03-12T02:54:03.227515+0000","last_unstale":"2024-03-12T02:54:31.002717+0000","last_undegraded":"2024-03-12T02:54:31.002717+0000","last_fullsized":"2024-03-12T02:54:31.002717+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:01:19.152969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'27","reported_seq":72,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006588+0000","last_change":"2024-03-12T02:54:05.235996+0000","last_active":"2024-03-12T02:54:31.006588+0000","last_peered":"2024-03-12T02:54:31.006588+0000","last_clean":"2024-03-12T02:54:31.006588+0000","last_became_active":"2024-03-12T02:54:05.235736+0000","last_became_peered":"2024-03-12T02:54:05.235736+0000","last_unstale":"2024-03-12T02:54:31.006588+0000","last_undegraded":"2024-03-12T02:54:31.006588+0000","last_fullsized":"2024-03-12T02:54:31.006588+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:19:13.386593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007890+0000","last_change":"2024-03-12T02:54:09.262476+0000","last_active":"2024-03-12T02:54:31.007890+0000","last_peered":"2024-03-12T02:54:31.007890+0000","last_clean":"2024-03-12T02:54:31.007890+0000","last_became_active":"2024-03-12T02:54:09.262219+0000","last_became_peered":"2024-03-12T02:54:09.262219+0000","last_unstale":"2024-03-12T02:54:31.007890+0000","last_undegraded":"2024-03-12T02:54:31.007890+0000","last_fullsized":"2024-03-12T02:54:31.007890+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:37.785243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008062+0000","last_change":"2024-03-12T02:54:03.225015+0000","last_active":"2024-03-12T02:54:31.008062+0000","last_peered":"2024-03-12T02:54:31.008062+0000","last_clean":"2024-03-12T02:54:31.008062+0000","last_became_active":"2024-03-12T02:54:03.223307+0000","last_became_peered":"2024-03-12T02:54:03.223307+0000","last_unstale":"2024-03-12T02:54:31.008062+0000","last_undegraded":"2024-03-12T02:54:31.008062+0000","last_fullsized":"2024-03-12T02:54:31.008062+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:40:25.332902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002923+0000","last_change":"2024-03-12T02:54:05.232779+0000","last_active":"2024-03-12T02:54:31.002923+0000","last_peered":"2024-03-12T02:54:31.002923+0000","last_clean":"2024-03-12T02:54:31.002923+0000","last_became_active":"2024-03-12T02:54:05.231956+0000","last_became_peered":"2024-03-12T02:54:05.231956+0000","last_unstale":"2024-03-12T02:54:31.002923+0000","last_undegraded":"2024-03-12T02:54:31.002923+0000","last_fullsized":"2024-03-12T02:54:31.002923+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:21:26.413572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005105+0000","last_change":"2024-03-12T02:54:07.255192+0000","last_active":"2024-03-12T02:54:31.005105+0000","last_peered":"2024-03-12T02:54:31.005105+0000","last_clean":"2024-03-12T02:54:31.005105+0000","last_became_active":"2024-03-12T02:54:07.254615+0000","last_became_peered":"2024-03-12T02:54:07.254615+0000","last_unstale":"2024-03-12T02:54:31.005105+0000","last_undegraded":"2024-03-12T02:54:31.005105+0000","last_fullsized":"2024-03-12T02:54:31.005105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:14:32.175047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003139+0000","last_change":"2024-03-12T02:54:09.261136+0000","last_active":"2024-03-12T02:54:31.003139+0000","last_peered":"2024-03-12T02:54:31.003139+0000","last_clean":"2024-03-12T02:54:31.003139+0000","last_became_active":"2024-03-12T02:54:09.260844+0000","last_became_peered":"2024-03-12T02:54:09.260844+0000","last_unstale":"2024-03-12T02:54:31.003139+0000","last_undegraded":"2024-03-12T02:54:31.003139+0000","last_fullsized":"2024-03-12T02:54:31.003139+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:53:17.706720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.275432+0000","last_change":"2024-03-12T02:54:07.256297+0000","last_active":"2024-03-12T02:54:33.275432+0000","last_peered":"2024-03-12T02:54:33.275432+0000","last_clean":"2024-03-12T02:54:33.275432+0000","last_became_active":"2024-03-12T02:54:07.255647+0000","last_became_peered":"2024-03-12T02:54:07.255647+0000","last_unstale":"2024-03-12T02:54:33.275432+0000","last_undegraded":"2024-03-12T02:54:33.275432+0000","last_fullsized":"2024-03-12T02:54:33.275432+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:40:13.586862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006870+0000","last_change":"2024-03-12T02:54:03.225714+0000","last_active":"2024-03-12T02:54:31.006870+0000","last_peered":"2024-03-12T02:54:31.006870+0000","last_clean":"2024-03-12T02:54:31.006870+0000","last_became_active":"2024-03-12T02:54:03.224262+0000","last_became_peered":"2024-03-12T02:54:03.224262+0000","last_unstale":"2024-03-12T02:54:31.006870+0000","last_undegraded":"2024-03-12T02:54:31.006870+0000","last_fullsized":"2024-03-12T02:54:31.006870+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:35:47.643035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'25","reported_seq":66,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004521+0000","last_change":"2024-03-12T02:54:05.237794+0000","last_active":"2024-03-12T02:54:31.004521+0000","last_peered":"2024-03-12T02:54:31.004521+0000","last_clean":"2024-03-12T02:54:31.004521+0000","last_became_active":"2024-03-12T02:54:05.237263+0000","last_became_peered":"2024-03-12T02:54:05.237263+0000","last_unstale":"2024-03-12T02:54:31.004521+0000","last_undegraded":"2024-03-12T02:54:31.004521+0000","last_fullsized":"2024-03-12T02:54:31.004521+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:01.667418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006552+0000","last_change":"2024-03-12T02:54:09.260348+0000","last_active":"2024-03-12T02:54:31.006552+0000","last_peered":"2024-03-12T02:54:31.006552+0000","last_clean":"2024-03-12T02:54:31.006552+0000","last_became_active":"2024-03-12T02:54:09.259332+0000","last_became_peered":"2024-03-12T02:54:09.259332+0000","last_unstale":"2024-03-12T02:54:31.006552+0000","last_undegraded":"2024-03-12T02:54:31.006552+0000","last_fullsized":"2024-03-12T02:54:31.006552+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:58:04.309965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007977+0000","last_change":"2024-03-12T02:54:03.222629+0000","last_active":"2024-03-12T02:54:31.007977+0000","last_peered":"2024-03-12T02:54:31.007977+0000","last_clean":"2024-03-12T02:54:31.007977+0000","last_became_active":"2024-03-12T02:54:03.222111+0000","last_became_peered":"2024-03-12T02:54:03.222111+0000","last_unstale":"2024-03-12T02:54:31.007977+0000","last_undegraded":"2024-03-12T02:54:31.007977+0000","last_fullsized":"2024-03-12T02:54:31.007977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:50:58.162282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006625+0000","last_change":"2024-03-12T02:54:05.237864+0000","last_active":"2024-03-12T02:54:31.006625+0000","last_peered":"2024-03-12T02:54:31.006625+0000","last_clean":"2024-03-12T02:54:31.006625+0000","last_became_active":"2024-03-12T02:54:05.237519+0000","last_became_peered":"2024-03-12T02:54:05.237519+0000","last_unstale":"2024-03-12T02:54:31.006625+0000","last_undegraded":"2024-03-12T02:54:31.006625+0000","last_fullsized":"2024-03-12T02:54:31.006625+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:27:54.863526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003998+0000","last_change":"2024-03-12T02:54:07.256148+0000","last_active":"2024-03-12T02:54:31.003998+0000","last_peered":"2024-03-12T02:54:31.003998+0000","last_clean":"2024-03-12T02:54:31.003998+0000","last_became_active":"2024-03-12T02:54:07.255589+0000","last_became_peered":"2024-03-12T02:54:07.255589+0000","last_unstale":"2024-03-12T02:54:31.003998+0000","last_undegraded":"2024-03-12T02:54:31.003998+0000","last_fullsized":"2024-03-12T02:54:31.003998+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:47:32.266619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.001965+0000","last_change":"2024-03-12T02:54:09.258391+0000","last_active":"2024-03-12T02:54:31.001965+0000","last_peered":"2024-03-12T02:54:31.001965+0000","last_clean":"2024-03-12T02:54:31.001965+0000","last_became_active":"2024-03-12T02:54:09.258041+0000","last_became_peered":"2024-03-12T02:54:09.258041+0000","last_unstale":"2024-03-12T02:54:31.001965+0000","last_undegraded":"2024-03-12T02:54:31.001965+0000","last_fullsized":"2024-03-12T02:54:31.001965+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:25:17.346673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008273+0000","last_change":"2024-03-12T02:54:03.224206+0000","last_active":"2024-03-12T02:54:31.008273+0000","last_peered":"2024-03-12T02:54:31.008273+0000","last_clean":"2024-03-12T02:54:31.008273+0000","last_became_active":"2024-03-12T02:54:03.223280+0000","last_became_peered":"2024-03-12T02:54:03.223280+0000","last_unstale":"2024-03-12T02:54:31.008273+0000","last_undegraded":"2024-03-12T02:54:31.008273+0000","last_fullsized":"2024-03-12T02:54:31.008273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:24.500603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005389+0000","last_change":"2024-03-12T02:54:05.238220+0000","last_active":"2024-03-12T02:54:31.005389+0000","last_peered":"2024-03-12T02:54:31.005389+0000","last_clean":"2024-03-12T02:54:31.005389+0000","last_became_active":"2024-03-12T02:54:05.237884+0000","last_became_peered":"2024-03-12T02:54:05.237884+0000","last_unstale":"2024-03-12T02:54:31.005389+0000","last_undegraded":"2024-03-12T02:54:31.005389+0000","last_fullsized":"2024-03-12T02:54:31.005389+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:17:41.716484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002459+0000","last_change":"2024-03-12T02:54:07.254060+0000","last_active":"2024-03-12T02:54:31.002459+0000","last_peered":"2024-03-12T02:54:31.002459+0000","last_clean":"2024-03-12T02:54:31.002459+0000","last_became_active":"2024-03-12T02:54:07.253067+0000","last_became_peered":"2024-03-12T02:54:07.253067+0000","last_unstale":"2024-03-12T02:54:31.002459+0000","last_undegraded":"2024-03-12T02:54:31.002459+0000","last_fullsized":"2024-03-12T02:54:31.002459+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:05.305908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003772+0000","last_change":"2024-03-12T02:54:09.263618+0000","last_active":"2024-03-12T02:54:31.003772+0000","last_peered":"2024-03-12T02:54:31.003772+0000","last_clean":"2024-03-12T02:54:31.003772+0000","last_became_active":"2024-03-12T02:54:09.262535+0000","last_became_peered":"2024-03-12T02:54:09.262535+0000","last_unstale":"2024-03-12T02:54:31.003772+0000","last_undegraded":"2024-03-12T02:54:31.003772+0000","last_fullsized":"2024-03-12T02:54:31.003772+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:55:18.615404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005061+0000","last_change":"2024-03-12T02:54:03.227748+0000","last_active":"2024-03-12T02:54:31.005061+0000","last_peered":"2024-03-12T02:54:31.005061+0000","last_clean":"2024-03-12T02:54:31.005061+0000","last_became_active":"2024-03-12T02:54:03.227486+0000","last_became_peered":"2024-03-12T02:54:03.227486+0000","last_unstale":"2024-03-12T02:54:31.005061+0000","last_undegraded":"2024-03-12T02:54:31.005061+0000","last_fullsized":"2024-03-12T02:54:31.005061+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:49:53.193904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006490+0000","last_change":"2024-03-12T02:54:05.238501+0000","last_active":"2024-03-12T02:54:31.006490+0000","last_peered":"2024-03-12T02:54:31.006490+0000","last_clean":"2024-03-12T02:54:31.006490+0000","last_became_active":"2024-03-12T02:54:05.238084+0000","last_became_peered":"2024-03-12T02:54:05.238084+0000","last_unstale":"2024-03-12T02:54:31.006490+0000","last_undegraded":"2024-03-12T02:54:31.006490+0000","last_fullsized":"2024-03-12T02:54:31.006490+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:05:02.591065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007210+0000","last_change":"2024-03-12T02:54:07.256290+0000","last_active":"2024-03-12T02:54:31.007210+0000","last_peered":"2024-03-12T02:54:31.007210+0000","last_clean":"2024-03-12T02:54:31.007210+0000","last_became_active":"2024-03-12T02:54:07.256011+0000","last_became_peered":"2024-03-12T02:54:07.256011+0000","last_unstale":"2024-03-12T02:54:31.007210+0000","last_undegraded":"2024-03-12T02:54:31.007210+0000","last_fullsized":"2024-03-12T02:54:31.007210+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:54:05.024023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":588,"ondisk_log_size":588,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411340,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738390,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T02:54:44.221 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T02:54:44.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:44 smithi079 ceph-mon[25794]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T02:54:44.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:44 smithi079 ceph-mon[25794]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T02:54:44.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:44 smithi115 ceph-mon[29819]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T02:54:44.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:44 smithi115 ceph-mon[29819]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T02:54:44.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:44 smithi129 ceph-mon[30321]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T02:54:44.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:44 smithi129 ceph-mon[30321]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T02:54:45.203 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:46.331 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:46 smithi079 ceph-mon[25794]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T02:54:46.331 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:46 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:54:46.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:46 smithi115 ceph-mon[29819]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T02:54:46.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:46 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:54:46.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:46 smithi129 ceph-mon[30321]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T02:54:46.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:46 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:54:47.718 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:54:47.720 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-03-12T02:54:48.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:48 smithi115 ceph-mon[29819]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T02:54:48.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:48 smithi115 ceph-mon[29819]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T02:54:48.475 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:48 smithi079 ceph-mon[25794]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T02:54:48.475 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:48 smithi079 ceph-mon[25794]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T02:54:48.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:48 smithi129 ceph-mon[30321]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T02:54:48.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:48 smithi129 ceph-mon[30321]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T02:54:48.900 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-03-12T02:54:47.015041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595984,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96636,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126532,"statfs":{"total":287972524032,"available":287873568768,"internally_reserved":0,"allocated":3182592,"data_stored":2109726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002342"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004573+0000","last_change":"2024-03-12T02:54:09.262852+0000","last_active":"2024-03-12T02:54:31.004573+0000","last_peered":"2024-03-12T02:54:31.004573+0000","last_clean":"2024-03-12T02:54:31.004573+0000","last_became_active":"2024-03-12T02:54:09.262475+0000","last_became_peered":"2024-03-12T02:54:09.262475+0000","last_unstale":"2024-03-12T02:54:31.004573+0000","last_undegraded":"2024-03-12T02:54:31.004573+0000","last_fullsized":"2024-03-12T02:54:31.004573+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:05:03.570464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006201+0000","last_change":"2024-03-12T02:54:07.256190+0000","last_active":"2024-03-12T02:54:31.006201+0000","last_peered":"2024-03-12T02:54:31.006201+0000","last_clean":"2024-03-12T02:54:31.006201+0000","last_became_active":"2024-03-12T02:54:07.255745+0000","last_became_peered":"2024-03-12T02:54:07.255745+0000","last_unstale":"2024-03-12T02:54:31.006201+0000","last_undegraded":"2024-03-12T02:54:31.006201+0000","last_fullsized":"2024-03-12T02:54:31.006201+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:15:33.801516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005851+0000","last_change":"2024-03-12T02:54:03.226988+0000","last_active":"2024-03-12T02:54:31.005851+0000","last_peered":"2024-03-12T02:54:31.005851+0000","last_clean":"2024-03-12T02:54:31.005851+0000","last_became_active":"2024-03-12T02:54:03.226722+0000","last_became_peered":"2024-03-12T02:54:03.226722+0000","last_unstale":"2024-03-12T02:54:31.005851+0000","last_undegraded":"2024-03-12T02:54:31.005851+0000","last_fullsized":"2024-03-12T02:54:31.005851+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:31:19.357681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'16","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005035+0000","last_change":"2024-03-12T02:54:05.235077+0000","last_active":"2024-03-12T02:54:31.005035+0000","last_peered":"2024-03-12T02:54:31.005035+0000","last_clean":"2024-03-12T02:54:31.005035+0000","last_became_active":"2024-03-12T02:54:05.233185+0000","last_became_peered":"2024-03-12T02:54:05.233185+0000","last_unstale":"2024-03-12T02:54:31.005035+0000","last_undegraded":"2024-03-12T02:54:31.005035+0000","last_fullsized":"2024-03-12T02:54:31.005035+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:10:08.885568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002691+0000","last_change":"2024-03-12T02:54:09.261727+0000","last_active":"2024-03-12T02:54:31.002691+0000","last_peered":"2024-03-12T02:54:31.002691+0000","last_clean":"2024-03-12T02:54:31.002691+0000","last_became_active":"2024-03-12T02:54:09.261355+0000","last_became_peered":"2024-03-12T02:54:09.261355+0000","last_unstale":"2024-03-12T02:54:31.002691+0000","last_undegraded":"2024-03-12T02:54:31.002691+0000","last_fullsized":"2024-03-12T02:54:31.002691+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:47.986881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004407+0000","last_change":"2024-03-12T02:54:07.257291+0000","last_active":"2024-03-12T02:54:31.004407+0000","last_peered":"2024-03-12T02:54:31.004407+0000","last_clean":"2024-03-12T02:54:31.004407+0000","last_became_active":"2024-03-12T02:54:07.256788+0000","last_became_peered":"2024-03-12T02:54:07.256788+0000","last_unstale":"2024-03-12T02:54:31.004407+0000","last_undegraded":"2024-03-12T02:54:31.004407+0000","last_fullsized":"2024-03-12T02:54:31.004407+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:34.754417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005320+0000","last_change":"2024-03-12T02:54:03.227646+0000","last_active":"2024-03-12T02:54:31.005320+0000","last_peered":"2024-03-12T02:54:31.005320+0000","last_clean":"2024-03-12T02:54:31.005320+0000","last_became_active":"2024-03-12T02:54:03.227360+0000","last_became_peered":"2024-03-12T02:54:03.227360+0000","last_unstale":"2024-03-12T02:54:31.005320+0000","last_undegraded":"2024-03-12T02:54:31.005320+0000","last_fullsized":"2024-03-12T02:54:31.005320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:18:25.747636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005600+0000","last_change":"2024-03-12T02:54:05.234431+0000","last_active":"2024-03-12T02:54:31.005600+0000","last_peered":"2024-03-12T02:54:31.005600+0000","last_clean":"2024-03-12T02:54:31.005600+0000","last_became_active":"2024-03-12T02:54:05.233052+0000","last_became_peered":"2024-03-12T02:54:05.233052+0000","last_unstale":"2024-03-12T02:54:31.005600+0000","last_undegraded":"2024-03-12T02:54:31.005600+0000","last_fullsized":"2024-03-12T02:54:31.005600+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:55:29.069618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006204+0000","last_change":"2024-03-12T02:54:09.261329+0000","last_active":"2024-03-12T02:54:31.006204+0000","last_peered":"2024-03-12T02:54:31.006204+0000","last_clean":"2024-03-12T02:54:31.006204+0000","last_became_active":"2024-03-12T02:54:09.261036+0000","last_became_peered":"2024-03-12T02:54:09.261036+0000","last_unstale":"2024-03-12T02:54:31.006204+0000","last_undegraded":"2024-03-12T02:54:31.006204+0000","last_fullsized":"2024-03-12T02:54:31.006204+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:15:38.555693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004561+0000","last_change":"2024-03-12T02:54:07.256265+0000","last_active":"2024-03-12T02:54:31.004561+0000","last_peered":"2024-03-12T02:54:31.004561+0000","last_clean":"2024-03-12T02:54:31.004561+0000","last_became_active":"2024-03-12T02:54:07.255716+0000","last_became_peered":"2024-03-12T02:54:07.255716+0000","last_unstale":"2024-03-12T02:54:31.004561+0000","last_undegraded":"2024-03-12T02:54:31.004561+0000","last_fullsized":"2024-03-12T02:54:31.004561+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:48:13.374889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.009809+0000","last_change":"2024-03-12T02:54:03.225600+0000","last_active":"2024-03-12T02:54:31.009809+0000","last_peered":"2024-03-12T02:54:31.009809+0000","last_clean":"2024-03-12T02:54:31.009809+0000","last_became_active":"2024-03-12T02:54:03.225188+0000","last_became_peered":"2024-03-12T02:54:03.225188+0000","last_unstale":"2024-03-12T02:54:31.009809+0000","last_undegraded":"2024-03-12T02:54:31.009809+0000","last_fullsized":"2024-03-12T02:54:31.009809+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:17:16.972224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005589+0000","last_change":"2024-03-12T02:54:05.237711+0000","last_active":"2024-03-12T02:54:31.005589+0000","last_peered":"2024-03-12T02:54:31.005589+0000","last_clean":"2024-03-12T02:54:31.005589+0000","last_became_active":"2024-03-12T02:54:05.237355+0000","last_became_peered":"2024-03-12T02:54:05.237355+0000","last_unstale":"2024-03-12T02:54:31.005589+0000","last_undegraded":"2024-03-12T02:54:31.005589+0000","last_fullsized":"2024-03-12T02:54:31.005589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:18:52.055675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004408+0000","last_change":"2024-03-12T02:54:09.256531+0000","last_active":"2024-03-12T02:54:31.004408+0000","last_peered":"2024-03-12T02:54:31.004408+0000","last_clean":"2024-03-12T02:54:31.004408+0000","last_became_active":"2024-03-12T02:54:09.256212+0000","last_became_peered":"2024-03-12T02:54:09.256212+0000","last_unstale":"2024-03-12T02:54:31.004408+0000","last_undegraded":"2024-03-12T02:54:31.004408+0000","last_fullsized":"2024-03-12T02:54:31.004408+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:27:46.690388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003871+0000","last_change":"2024-03-12T02:54:07.257143+0000","last_active":"2024-03-12T02:54:31.003871+0000","last_peered":"2024-03-12T02:54:31.003871+0000","last_clean":"2024-03-12T02:54:31.003871+0000","last_became_active":"2024-03-12T02:54:07.256713+0000","last_became_peered":"2024-03-12T02:54:07.256713+0000","last_unstale":"2024-03-12T02:54:31.003871+0000","last_undegraded":"2024-03-12T02:54:31.003871+0000","last_fullsized":"2024-03-12T02:54:31.003871+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:03:45.354354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008080+0000","last_change":"2024-03-12T02:54:03.225555+0000","last_active":"2024-03-12T02:54:31.008080+0000","last_peered":"2024-03-12T02:54:31.008080+0000","last_clean":"2024-03-12T02:54:31.008080+0000","last_became_active":"2024-03-12T02:54:03.223730+0000","last_became_peered":"2024-03-12T02:54:03.223730+0000","last_unstale":"2024-03-12T02:54:31.008080+0000","last_undegraded":"2024-03-12T02:54:31.008080+0000","last_fullsized":"2024-03-12T02:54:31.008080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:48:38.603242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.009720+0000","last_change":"2024-03-12T02:54:05.236261+0000","last_active":"2024-03-12T02:54:31.009720+0000","last_peered":"2024-03-12T02:54:31.009720+0000","last_clean":"2024-03-12T02:54:31.009720+0000","last_became_active":"2024-03-12T02:54:05.235912+0000","last_became_peered":"2024-03-12T02:54:05.235912+0000","last_unstale":"2024-03-12T02:54:31.009720+0000","last_undegraded":"2024-03-12T02:54:31.009720+0000","last_fullsized":"2024-03-12T02:54:31.009720+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:32:30.495642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004989+0000","last_change":"2024-03-12T02:54:09.256454+0000","last_active":"2024-03-12T02:54:31.004989+0000","last_peered":"2024-03-12T02:54:31.004989+0000","last_clean":"2024-03-12T02:54:31.004989+0000","last_became_active":"2024-03-12T02:54:09.256084+0000","last_became_peered":"2024-03-12T02:54:09.256084+0000","last_unstale":"2024-03-12T02:54:31.004989+0000","last_undegraded":"2024-03-12T02:54:31.004989+0000","last_fullsized":"2024-03-12T02:54:31.004989+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:09:45.979440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008011+0000","last_change":"2024-03-12T02:54:07.251840+0000","last_active":"2024-03-12T02:54:31.008011+0000","last_peered":"2024-03-12T02:54:31.008011+0000","last_clean":"2024-03-12T02:54:31.008011+0000","last_became_active":"2024-03-12T02:54:07.251524+0000","last_became_peered":"2024-03-12T02:54:07.251524+0000","last_unstale":"2024-03-12T02:54:31.008011+0000","last_undegraded":"2024-03-12T02:54:31.008011+0000","last_fullsized":"2024-03-12T02:54:31.008011+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:27:15.892746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008080+0000","last_change":"2024-03-12T02:54:03.224079+0000","last_active":"2024-03-12T02:54:31.008080+0000","last_peered":"2024-03-12T02:54:31.008080+0000","last_clean":"2024-03-12T02:54:31.008080+0000","last_became_active":"2024-03-12T02:54:03.223147+0000","last_became_peered":"2024-03-12T02:54:03.223147+0000","last_unstale":"2024-03-12T02:54:31.008080+0000","last_undegraded":"2024-03-12T02:54:31.008080+0000","last_fullsized":"2024-03-12T02:54:31.008080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:05:47.427879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'23","reported_seq":65,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008158+0000","last_change":"2024-03-12T02:54:05.234670+0000","last_active":"2024-03-12T02:54:31.008158+0000","last_peered":"2024-03-12T02:54:31.008158+0000","last_clean":"2024-03-12T02:54:31.008158+0000","last_became_active":"2024-03-12T02:54:05.234310+0000","last_became_peered":"2024-03-12T02:54:05.234310+0000","last_unstale":"2024-03-12T02:54:31.008158+0000","last_undegraded":"2024-03-12T02:54:31.008158+0000","last_fullsized":"2024-03-12T02:54:31.008158+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:01:50.146305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004616+0000","last_change":"2024-03-12T02:54:09.261629+0000","last_active":"2024-03-12T02:54:31.004616+0000","last_peered":"2024-03-12T02:54:31.004616+0000","last_clean":"2024-03-12T02:54:31.004616+0000","last_became_active":"2024-03-12T02:54:09.261334+0000","last_became_peered":"2024-03-12T02:54:09.261334+0000","last_unstale":"2024-03-12T02:54:31.004616+0000","last_undegraded":"2024-03-12T02:54:31.004616+0000","last_fullsized":"2024-03-12T02:54:31.004616+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:34:56.643839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007849+0000","last_change":"2024-03-12T02:54:07.255272+0000","last_active":"2024-03-12T02:54:31.007849+0000","last_peered":"2024-03-12T02:54:31.007849+0000","last_clean":"2024-03-12T02:54:31.007849+0000","last_became_active":"2024-03-12T02:54:07.254979+0000","last_became_peered":"2024-03-12T02:54:07.254979+0000","last_unstale":"2024-03-12T02:54:31.007849+0000","last_undegraded":"2024-03-12T02:54:31.007849+0000","last_fullsized":"2024-03-12T02:54:31.007849+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:49.219851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004296+0000","last_change":"2024-03-12T02:54:03.221816+0000","last_active":"2024-03-12T02:54:31.004296+0000","last_peered":"2024-03-12T02:54:31.004296+0000","last_clean":"2024-03-12T02:54:31.004296+0000","last_became_active":"2024-03-12T02:54:03.221022+0000","last_became_peered":"2024-03-12T02:54:03.221022+0000","last_unstale":"2024-03-12T02:54:31.004296+0000","last_undegraded":"2024-03-12T02:54:31.004296+0000","last_fullsized":"2024-03-12T02:54:31.004296+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:39:50.080867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'11","reported_seq":45,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007405+0000","last_change":"2024-03-12T02:54:05.234573+0000","last_active":"2024-03-12T02:54:31.007405+0000","last_peered":"2024-03-12T02:54:31.007405+0000","last_clean":"2024-03-12T02:54:31.007405+0000","last_became_active":"2024-03-12T02:54:05.234055+0000","last_became_peered":"2024-03-12T02:54:05.234055+0000","last_unstale":"2024-03-12T02:54:31.007405+0000","last_undegraded":"2024-03-12T02:54:31.007405+0000","last_fullsized":"2024-03-12T02:54:31.007405+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:20.329923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004359+0000","last_change":"2024-03-12T02:54:09.258576+0000","last_active":"2024-03-12T02:54:31.004359+0000","last_peered":"2024-03-12T02:54:31.004359+0000","last_clean":"2024-03-12T02:54:31.004359+0000","last_became_active":"2024-03-12T02:54:09.257789+0000","last_became_peered":"2024-03-12T02:54:09.257789+0000","last_unstale":"2024-03-12T02:54:31.004359+0000","last_undegraded":"2024-03-12T02:54:31.004359+0000","last_fullsized":"2024-03-12T02:54:31.004359+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:10:02.425696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:38.269758+0000","last_change":"2024-03-12T02:54:07.253050+0000","last_active":"2024-03-12T02:54:38.269758+0000","last_peered":"2024-03-12T02:54:38.269758+0000","last_clean":"2024-03-12T02:54:38.269758+0000","last_became_active":"2024-03-12T02:54:07.252317+0000","last_became_peered":"2024-03-12T02:54:07.252317+0000","last_unstale":"2024-03-12T02:54:38.269758+0000","last_undegraded":"2024-03-12T02:54:38.269758+0000","last_fullsized":"2024-03-12T02:54:38.269758+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:38.947170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002857+0000","last_change":"2024-03-12T02:54:03.227812+0000","last_active":"2024-03-12T02:54:31.002857+0000","last_peered":"2024-03-12T02:54:31.002857+0000","last_clean":"2024-03-12T02:54:31.002857+0000","last_became_active":"2024-03-12T02:54:03.227429+0000","last_became_peered":"2024-03-12T02:54:03.227429+0000","last_unstale":"2024-03-12T02:54:31.002857+0000","last_undegraded":"2024-03-12T02:54:31.002857+0000","last_fullsized":"2024-03-12T02:54:31.002857+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:07.072163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.001939+0000","last_change":"2024-03-12T02:54:05.237039+0000","last_active":"2024-03-12T02:54:31.001939+0000","last_peered":"2024-03-12T02:54:31.001939+0000","last_clean":"2024-03-12T02:54:31.001939+0000","last_became_active":"2024-03-12T02:54:05.234567+0000","last_became_peered":"2024-03-12T02:54:05.234567+0000","last_unstale":"2024-03-12T02:54:31.001939+0000","last_undegraded":"2024-03-12T02:54:31.001939+0000","last_fullsized":"2024-03-12T02:54:31.001939+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:22:26.866756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005289+0000","last_change":"2024-03-12T02:54:09.263222+0000","last_active":"2024-03-12T02:54:31.005289+0000","last_peered":"2024-03-12T02:54:31.005289+0000","last_clean":"2024-03-12T02:54:31.005289+0000","last_became_active":"2024-03-12T02:54:09.262639+0000","last_became_peered":"2024-03-12T02:54:09.262639+0000","last_unstale":"2024-03-12T02:54:31.005289+0000","last_undegraded":"2024-03-12T02:54:31.005289+0000","last_fullsized":"2024-03-12T02:54:31.005289+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:34:53.666069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004523+0000","last_change":"2024-03-12T02:54:07.259689+0000","last_active":"2024-03-12T02:54:31.004523+0000","last_peered":"2024-03-12T02:54:31.004523+0000","last_clean":"2024-03-12T02:54:31.004523+0000","last_became_active":"2024-03-12T02:54:07.259373+0000","last_became_peered":"2024-03-12T02:54:07.259373+0000","last_unstale":"2024-03-12T02:54:31.004523+0000","last_undegraded":"2024-03-12T02:54:31.004523+0000","last_fullsized":"2024-03-12T02:54:31.004523+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:14:23.466948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006396+0000","last_change":"2024-03-12T02:54:03.222198+0000","last_active":"2024-03-12T02:54:31.006396+0000","last_peered":"2024-03-12T02:54:31.006396+0000","last_clean":"2024-03-12T02:54:31.006396+0000","last_became_active":"2024-03-12T02:54:03.221869+0000","last_became_peered":"2024-03-12T02:54:03.221869+0000","last_unstale":"2024-03-12T02:54:31.006396+0000","last_undegraded":"2024-03-12T02:54:31.006396+0000","last_fullsized":"2024-03-12T02:54:31.006396+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:50:58.355917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'39","reported_seq":96,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002358+0000","last_change":"2024-03-12T02:54:05.232425+0000","last_active":"2024-03-12T02:54:31.002358+0000","last_peered":"2024-03-12T02:54:31.002358+0000","last_clean":"2024-03-12T02:54:31.002358+0000","last_became_active":"2024-03-12T02:54:05.231864+0000","last_became_peered":"2024-03-12T02:54:05.231864+0000","last_unstale":"2024-03-12T02:54:31.002358+0000","last_undegraded":"2024-03-12T02:54:31.002358+0000","last_fullsized":"2024-03-12T02:54:31.002358+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:34:57.218306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004708+0000","last_change":"2024-03-12T02:54:09.264184+0000","last_active":"2024-03-12T02:54:31.004708+0000","last_peered":"2024-03-12T02:54:31.004708+0000","last_clean":"2024-03-12T02:54:31.004708+0000","last_became_active":"2024-03-12T02:54:09.263684+0000","last_became_peered":"2024-03-12T02:54:09.263684+0000","last_unstale":"2024-03-12T02:54:31.004708+0000","last_undegraded":"2024-03-12T02:54:31.004708+0000","last_fullsized":"2024-03-12T02:54:31.004708+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:10:20.734586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.001779+0000","last_change":"2024-03-12T02:54:07.253843+0000","last_active":"2024-03-12T02:54:31.001779+0000","last_peered":"2024-03-12T02:54:31.001779+0000","last_clean":"2024-03-12T02:54:31.001779+0000","last_became_active":"2024-03-12T02:54:07.252732+0000","last_became_peered":"2024-03-12T02:54:07.252732+0000","last_unstale":"2024-03-12T02:54:31.001779+0000","last_undegraded":"2024-03-12T02:54:31.001779+0000","last_fullsized":"2024-03-12T02:54:31.001779+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:35:23.782833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006630+0000","last_change":"2024-03-12T02:54:03.228617+0000","last_active":"2024-03-12T02:54:31.006630+0000","last_peered":"2024-03-12T02:54:31.006630+0000","last_clean":"2024-03-12T02:54:31.006630+0000","last_became_active":"2024-03-12T02:54:03.228302+0000","last_became_peered":"2024-03-12T02:54:03.228302+0000","last_unstale":"2024-03-12T02:54:31.006630+0000","last_undegraded":"2024-03-12T02:54:31.006630+0000","last_fullsized":"2024-03-12T02:54:31.006630+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:35:20.881427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'21","reported_seq":63,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006822+0000","last_change":"2024-03-12T02:54:05.238673+0000","last_active":"2024-03-12T02:54:31.006822+0000","last_peered":"2024-03-12T02:54:31.006822+0000","last_clean":"2024-03-12T02:54:31.006822+0000","last_became_active":"2024-03-12T02:54:05.238341+0000","last_became_peered":"2024-03-12T02:54:05.238341+0000","last_unstale":"2024-03-12T02:54:31.006822+0000","last_undegraded":"2024-03-12T02:54:31.006822+0000","last_fullsized":"2024-03-12T02:54:31.006822+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:32:03.077399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004055+0000","last_change":"2024-03-12T02:54:09.263320+0000","last_active":"2024-03-12T02:54:31.004055+0000","last_peered":"2024-03-12T02:54:31.004055+0000","last_clean":"2024-03-12T02:54:31.004055+0000","last_became_active":"2024-03-12T02:54:09.262528+0000","last_became_peered":"2024-03-12T02:54:09.262528+0000","last_unstale":"2024-03-12T02:54:31.004055+0000","last_undegraded":"2024-03-12T02:54:31.004055+0000","last_fullsized":"2024-03-12T02:54:31.004055+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:29:22.197950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002020+0000","last_change":"2024-03-12T02:54:07.256470+0000","last_active":"2024-03-12T02:54:31.002020+0000","last_peered":"2024-03-12T02:54:31.002020+0000","last_clean":"2024-03-12T02:54:31.002020+0000","last_became_active":"2024-03-12T02:54:07.256135+0000","last_became_peered":"2024-03-12T02:54:07.256135+0000","last_unstale":"2024-03-12T02:54:31.002020+0000","last_undegraded":"2024-03-12T02:54:31.002020+0000","last_fullsized":"2024-03-12T02:54:31.002020+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:34:43.356196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006572+0000","last_change":"2024-03-12T02:54:03.227079+0000","last_active":"2024-03-12T02:54:31.006572+0000","last_peered":"2024-03-12T02:54:31.006572+0000","last_clean":"2024-03-12T02:54:31.006572+0000","last_became_active":"2024-03-12T02:54:03.226635+0000","last_became_peered":"2024-03-12T02:54:03.226635+0000","last_unstale":"2024-03-12T02:54:31.006572+0000","last_undegraded":"2024-03-12T02:54:31.006572+0000","last_fullsized":"2024-03-12T02:54:31.006572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:36:29.069294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'12","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005729+0000","last_change":"2024-03-12T02:54:05.235495+0000","last_active":"2024-03-12T02:54:31.005729+0000","last_peered":"2024-03-12T02:54:31.005729+0000","last_clean":"2024-03-12T02:54:31.005729+0000","last_became_active":"2024-03-12T02:54:05.235162+0000","last_became_peered":"2024-03-12T02:54:05.235162+0000","last_unstale":"2024-03-12T02:54:31.005729+0000","last_undegraded":"2024-03-12T02:54:31.005729+0000","last_fullsized":"2024-03-12T02:54:31.005729+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:26:08.135711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006390+0000","last_change":"2024-03-12T02:54:09.262079+0000","last_active":"2024-03-12T02:54:31.006390+0000","last_peered":"2024-03-12T02:54:31.006390+0000","last_clean":"2024-03-12T02:54:31.006390+0000","last_became_active":"2024-03-12T02:54:09.261319+0000","last_became_peered":"2024-03-12T02:54:09.261319+0000","last_unstale":"2024-03-12T02:54:31.006390+0000","last_undegraded":"2024-03-12T02:54:31.006390+0000","last_fullsized":"2024-03-12T02:54:31.006390+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:59:00.259665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004661+0000","last_change":"2024-03-12T02:54:07.255224+0000","last_active":"2024-03-12T02:54:31.004661+0000","last_peered":"2024-03-12T02:54:31.004661+0000","last_clean":"2024-03-12T02:54:31.004661+0000","last_became_active":"2024-03-12T02:54:07.254656+0000","last_became_peered":"2024-03-12T02:54:07.254656+0000","last_unstale":"2024-03-12T02:54:31.004661+0000","last_undegraded":"2024-03-12T02:54:31.004661+0000","last_fullsized":"2024-03-12T02:54:31.004661+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:51:47.700475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008721+0000","last_change":"2024-03-12T02:54:03.223436+0000","last_active":"2024-03-12T02:54:31.008721+0000","last_peered":"2024-03-12T02:54:31.008721+0000","last_clean":"2024-03-12T02:54:31.008721+0000","last_became_active":"2024-03-12T02:54:03.222345+0000","last_became_peered":"2024-03-12T02:54:03.222345+0000","last_unstale":"2024-03-12T02:54:31.008721+0000","last_undegraded":"2024-03-12T02:54:31.008721+0000","last_fullsized":"2024-03-12T02:54:31.008721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:54:18.905498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'22","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005664+0000","last_change":"2024-03-12T02:54:05.238022+0000","last_active":"2024-03-12T02:54:31.005664+0000","last_peered":"2024-03-12T02:54:31.005664+0000","last_clean":"2024-03-12T02:54:31.005664+0000","last_became_active":"2024-03-12T02:54:05.237356+0000","last_became_peered":"2024-03-12T02:54:05.237356+0000","last_unstale":"2024-03-12T02:54:31.005664+0000","last_undegraded":"2024-03-12T02:54:31.005664+0000","last_fullsized":"2024-03-12T02:54:31.005664+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:23:57.767122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006609+0000","last_change":"2024-03-12T02:54:09.261304+0000","last_active":"2024-03-12T02:54:31.006609+0000","last_peered":"2024-03-12T02:54:31.006609+0000","last_clean":"2024-03-12T02:54:31.006609+0000","last_became_active":"2024-03-12T02:54:09.261004+0000","last_became_peered":"2024-03-12T02:54:09.261004+0000","last_unstale":"2024-03-12T02:54:31.006609+0000","last_undegraded":"2024-03-12T02:54:31.006609+0000","last_fullsized":"2024-03-12T02:54:31.006609+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:17:36.305661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002158+0000","last_change":"2024-03-12T02:54:07.254115+0000","last_active":"2024-03-12T02:54:31.002158+0000","last_peered":"2024-03-12T02:54:31.002158+0000","last_clean":"2024-03-12T02:54:31.002158+0000","last_became_active":"2024-03-12T02:54:07.253729+0000","last_became_peered":"2024-03-12T02:54:07.253729+0000","last_unstale":"2024-03-12T02:54:31.002158+0000","last_undegraded":"2024-03-12T02:54:31.002158+0000","last_fullsized":"2024-03-12T02:54:31.002158+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:44:09.014023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005306+0000","last_change":"2024-03-12T02:54:03.227350+0000","last_active":"2024-03-12T02:54:31.005306+0000","last_peered":"2024-03-12T02:54:31.005306+0000","last_clean":"2024-03-12T02:54:31.005306+0000","last_became_active":"2024-03-12T02:54:03.226910+0000","last_became_peered":"2024-03-12T02:54:03.226910+0000","last_unstale":"2024-03-12T02:54:31.005306+0000","last_undegraded":"2024-03-12T02:54:31.005306+0000","last_fullsized":"2024-03-12T02:54:31.005306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:10:21.918099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008279+0000","last_change":"2024-03-12T02:54:05.234490+0000","last_active":"2024-03-12T02:54:31.008279+0000","last_peered":"2024-03-12T02:54:31.008279+0000","last_clean":"2024-03-12T02:54:31.008279+0000","last_became_active":"2024-03-12T02:54:05.234122+0000","last_became_peered":"2024-03-12T02:54:05.234122+0000","last_unstale":"2024-03-12T02:54:31.008279+0000","last_undegraded":"2024-03-12T02:54:31.008279+0000","last_fullsized":"2024-03-12T02:54:31.008279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:10:20.714956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003405+0000","last_change":"2024-03-12T02:54:09.258361+0000","last_active":"2024-03-12T02:54:31.003405+0000","last_peered":"2024-03-12T02:54:31.003405+0000","last_clean":"2024-03-12T02:54:31.003405+0000","last_became_active":"2024-03-12T02:54:09.257929+0000","last_became_peered":"2024-03-12T02:54:09.257929+0000","last_unstale":"2024-03-12T02:54:31.003405+0000","last_undegraded":"2024-03-12T02:54:31.003405+0000","last_fullsized":"2024-03-12T02:54:31.003405+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:14:21.972211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:38.269769+0000","last_change":"2024-03-12T02:54:07.252459+0000","last_active":"2024-03-12T02:54:38.269769+0000","last_peered":"2024-03-12T02:54:38.269769+0000","last_clean":"2024-03-12T02:54:38.269769+0000","last_became_active":"2024-03-12T02:54:07.252184+0000","last_became_peered":"2024-03-12T02:54:07.252184+0000","last_unstale":"2024-03-12T02:54:38.269769+0000","last_undegraded":"2024-03-12T02:54:38.269769+0000","last_fullsized":"2024-03-12T02:54:38.269769+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:29.703459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007045+0000","last_change":"2024-03-12T02:54:03.223425+0000","last_active":"2024-03-12T02:54:31.007045+0000","last_peered":"2024-03-12T02:54:31.007045+0000","last_clean":"2024-03-12T02:54:31.007045+0000","last_became_active":"2024-03-12T02:54:03.222484+0000","last_became_peered":"2024-03-12T02:54:03.222484+0000","last_unstale":"2024-03-12T02:54:31.007045+0000","last_undegraded":"2024-03-12T02:54:31.007045+0000","last_fullsized":"2024-03-12T02:54:31.007045+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:32:32.455804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002503+0000","last_change":"2024-03-12T02:54:05.232774+0000","last_active":"2024-03-12T02:54:31.002503+0000","last_peered":"2024-03-12T02:54:31.002503+0000","last_clean":"2024-03-12T02:54:31.002503+0000","last_became_active":"2024-03-12T02:54:05.232449+0000","last_became_peered":"2024-03-12T02:54:05.232449+0000","last_unstale":"2024-03-12T02:54:31.002503+0000","last_undegraded":"2024-03-12T02:54:31.002503+0000","last_fullsized":"2024-03-12T02:54:31.002503+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:03:08.187506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005588+0000","last_change":"2024-03-12T02:54:09.264192+0000","last_active":"2024-03-12T02:54:31.005588+0000","last_peered":"2024-03-12T02:54:31.005588+0000","last_clean":"2024-03-12T02:54:31.005588+0000","last_became_active":"2024-03-12T02:54:09.263557+0000","last_became_peered":"2024-03-12T02:54:09.263557+0000","last_unstale":"2024-03-12T02:54:31.005588+0000","last_undegraded":"2024-03-12T02:54:31.005588+0000","last_fullsized":"2024-03-12T02:54:31.005588+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:27.630121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:38.269770+0000","last_change":"2024-03-12T02:54:07.255094+0000","last_active":"2024-03-12T02:54:38.269770+0000","last_peered":"2024-03-12T02:54:38.269770+0000","last_clean":"2024-03-12T02:54:38.269770+0000","last_became_active":"2024-03-12T02:54:07.254776+0000","last_became_peered":"2024-03-12T02:54:07.254776+0000","last_unstale":"2024-03-12T02:54:38.269770+0000","last_undegraded":"2024-03-12T02:54:38.269770+0000","last_fullsized":"2024-03-12T02:54:38.269770+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:34:04.511468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006601+0000","last_change":"2024-03-12T02:54:03.225046+0000","last_active":"2024-03-12T02:54:31.006601+0000","last_peered":"2024-03-12T02:54:31.006601+0000","last_clean":"2024-03-12T02:54:31.006601+0000","last_became_active":"2024-03-12T02:54:03.223354+0000","last_became_peered":"2024-03-12T02:54:03.223354+0000","last_unstale":"2024-03-12T02:54:31.006601+0000","last_undegraded":"2024-03-12T02:54:31.006601+0000","last_fullsized":"2024-03-12T02:54:31.006601+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:30:33.792780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004911+0000","last_change":"2024-03-12T02:54:05.236021+0000","last_active":"2024-03-12T02:54:31.004911+0000","last_peered":"2024-03-12T02:54:31.004911+0000","last_clean":"2024-03-12T02:54:31.004911+0000","last_became_active":"2024-03-12T02:54:05.235517+0000","last_became_peered":"2024-03-12T02:54:05.235517+0000","last_unstale":"2024-03-12T02:54:31.004911+0000","last_undegraded":"2024-03-12T02:54:31.004911+0000","last_fullsized":"2024-03-12T02:54:31.004911+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:46.468603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005907+0000","last_change":"2024-03-12T02:54:09.257663+0000","last_active":"2024-03-12T02:54:31.005907+0000","last_peered":"2024-03-12T02:54:31.005907+0000","last_clean":"2024-03-12T02:54:31.005907+0000","last_became_active":"2024-03-12T02:54:09.257321+0000","last_became_peered":"2024-03-12T02:54:09.257321+0000","last_unstale":"2024-03-12T02:54:31.005907+0000","last_undegraded":"2024-03-12T02:54:31.005907+0000","last_fullsized":"2024-03-12T02:54:31.005907+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:19:25.103143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005489+0000","last_change":"2024-03-12T02:54:07.259511+0000","last_active":"2024-03-12T02:54:31.005489+0000","last_peered":"2024-03-12T02:54:31.005489+0000","last_clean":"2024-03-12T02:54:31.005489+0000","last_became_active":"2024-03-12T02:54:07.259200+0000","last_became_peered":"2024-03-12T02:54:07.259200+0000","last_unstale":"2024-03-12T02:54:31.005489+0000","last_undegraded":"2024-03-12T02:54:31.005489+0000","last_fullsized":"2024-03-12T02:54:31.005489+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:23:48.027202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003989+0000","last_change":"2024-03-12T02:54:03.221775+0000","last_active":"2024-03-12T02:54:31.003989+0000","last_peered":"2024-03-12T02:54:31.003989+0000","last_clean":"2024-03-12T02:54:31.003989+0000","last_became_active":"2024-03-12T02:54:03.220660+0000","last_became_peered":"2024-03-12T02:54:03.220660+0000","last_unstale":"2024-03-12T02:54:31.003989+0000","last_undegraded":"2024-03-12T02:54:31.003989+0000","last_fullsized":"2024-03-12T02:54:31.003989+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:02:07.511766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004122+0000","last_change":"2024-03-12T02:54:05.238734+0000","last_active":"2024-03-12T02:54:31.004122+0000","last_peered":"2024-03-12T02:54:31.004122+0000","last_clean":"2024-03-12T02:54:31.004122+0000","last_became_active":"2024-03-12T02:54:05.238417+0000","last_became_peered":"2024-03-12T02:54:05.238417+0000","last_unstale":"2024-03-12T02:54:31.004122+0000","last_undegraded":"2024-03-12T02:54:31.004122+0000","last_fullsized":"2024-03-12T02:54:31.004122+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:23:18.321847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005231+0000","last_change":"2024-03-12T02:54:09.261271+0000","last_active":"2024-03-12T02:54:31.005231+0000","last_peered":"2024-03-12T02:54:31.005231+0000","last_clean":"2024-03-12T02:54:31.005231+0000","last_became_active":"2024-03-12T02:54:09.260913+0000","last_became_peered":"2024-03-12T02:54:09.260913+0000","last_unstale":"2024-03-12T02:54:31.005231+0000","last_undegraded":"2024-03-12T02:54:31.005231+0000","last_fullsized":"2024-03-12T02:54:31.005231+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:50:53.351133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004610+0000","last_change":"2024-03-12T02:54:07.259642+0000","last_active":"2024-03-12T02:54:31.004610+0000","last_peered":"2024-03-12T02:54:31.004610+0000","last_clean":"2024-03-12T02:54:31.004610+0000","last_became_active":"2024-03-12T02:54:07.259240+0000","last_became_peered":"2024-03-12T02:54:07.259240+0000","last_unstale":"2024-03-12T02:54:31.004610+0000","last_undegraded":"2024-03-12T02:54:31.004610+0000","last_fullsized":"2024-03-12T02:54:31.004610+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:55:16.464794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006498+0000","last_change":"2024-03-12T02:54:03.223127+0000","last_active":"2024-03-12T02:54:31.006498+0000","last_peered":"2024-03-12T02:54:31.006498+0000","last_clean":"2024-03-12T02:54:31.006498+0000","last_became_active":"2024-03-12T02:54:03.222068+0000","last_became_peered":"2024-03-12T02:54:03.222068+0000","last_unstale":"2024-03-12T02:54:31.006498+0000","last_undegraded":"2024-03-12T02:54:31.006498+0000","last_fullsized":"2024-03-12T02:54:31.006498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:59:01.816434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007235+0000","last_change":"2024-03-12T02:54:05.236063+0000","last_active":"2024-03-12T02:54:31.007235+0000","last_peered":"2024-03-12T02:54:31.007235+0000","last_clean":"2024-03-12T02:54:31.007235+0000","last_became_active":"2024-03-12T02:54:05.235767+0000","last_became_peered":"2024-03-12T02:54:05.235767+0000","last_unstale":"2024-03-12T02:54:31.007235+0000","last_undegraded":"2024-03-12T02:54:31.007235+0000","last_fullsized":"2024-03-12T02:54:31.007235+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:35:34.341194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007302+0000","last_change":"2024-03-12T02:54:09.261429+0000","last_active":"2024-03-12T02:54:31.007302+0000","last_peered":"2024-03-12T02:54:31.007302+0000","last_clean":"2024-03-12T02:54:31.007302+0000","last_became_active":"2024-03-12T02:54:09.261104+0000","last_became_peered":"2024-03-12T02:54:09.261104+0000","last_unstale":"2024-03-12T02:54:31.007302+0000","last_undegraded":"2024-03-12T02:54:31.007302+0000","last_fullsized":"2024-03-12T02:54:31.007302+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:23:04.807606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008651+0000","last_change":"2024-03-12T02:54:03.225515+0000","last_active":"2024-03-12T02:54:31.008651+0000","last_peered":"2024-03-12T02:54:31.008651+0000","last_clean":"2024-03-12T02:54:31.008651+0000","last_became_active":"2024-03-12T02:54:03.223339+0000","last_became_peered":"2024-03-12T02:54:03.223339+0000","last_unstale":"2024-03-12T02:54:31.008651+0000","last_undegraded":"2024-03-12T02:54:31.008651+0000","last_fullsized":"2024-03-12T02:54:31.008651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:25.270252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'17","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008711+0000","last_change":"2024-03-12T02:54:05.236067+0000","last_active":"2024-03-12T02:54:31.008711+0000","last_peered":"2024-03-12T02:54:31.008711+0000","last_clean":"2024-03-12T02:54:31.008711+0000","last_became_active":"2024-03-12T02:54:05.235692+0000","last_became_peered":"2024-03-12T02:54:05.235692+0000","last_unstale":"2024-03-12T02:54:31.008711+0000","last_undegraded":"2024-03-12T02:54:31.008711+0000","last_fullsized":"2024-03-12T02:54:31.008711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:50.573680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:38.269808+0000","last_change":"2024-03-12T02:54:07.255405+0000","last_active":"2024-03-12T02:54:38.269808+0000","last_peered":"2024-03-12T02:54:38.269808+0000","last_clean":"2024-03-12T02:54:38.269808+0000","last_became_active":"2024-03-12T02:54:07.255072+0000","last_became_peered":"2024-03-12T02:54:07.255072+0000","last_unstale":"2024-03-12T02:54:38.269808+0000","last_undegraded":"2024-03-12T02:54:38.269808+0000","last_fullsized":"2024-03-12T02:54:38.269808+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:50:43.443544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002347+0000","last_change":"2024-03-12T02:54:09.258261+0000","last_active":"2024-03-12T02:54:31.002347+0000","last_peered":"2024-03-12T02:54:31.002347+0000","last_clean":"2024-03-12T02:54:31.002347+0000","last_became_active":"2024-03-12T02:54:09.257861+0000","last_became_peered":"2024-03-12T02:54:09.257861+0000","last_unstale":"2024-03-12T02:54:31.002347+0000","last_undegraded":"2024-03-12T02:54:31.002347+0000","last_fullsized":"2024-03-12T02:54:31.002347+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:32:05.076028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004134+0000","last_change":"2024-03-12T02:54:03.228357+0000","last_active":"2024-03-12T02:54:31.004134+0000","last_peered":"2024-03-12T02:54:31.004134+0000","last_clean":"2024-03-12T02:54:31.004134+0000","last_became_active":"2024-03-12T02:54:03.228003+0000","last_became_peered":"2024-03-12T02:54:03.228003+0000","last_unstale":"2024-03-12T02:54:31.004134+0000","last_undegraded":"2024-03-12T02:54:31.004134+0000","last_fullsized":"2024-03-12T02:54:31.004134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:42:28.054205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:38.269622+0000","last_change":"2024-03-12T02:54:05.236101+0000","last_active":"2024-03-12T02:54:38.269622+0000","last_peered":"2024-03-12T02:54:38.269622+0000","last_clean":"2024-03-12T02:54:38.269622+0000","last_became_active":"2024-03-12T02:54:05.235549+0000","last_became_peered":"2024-03-12T02:54:05.235549+0000","last_unstale":"2024-03-12T02:54:38.269622+0000","last_undegraded":"2024-03-12T02:54:38.269622+0000","last_fullsized":"2024-03-12T02:54:38.269622+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:27:39.230654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006419+0000","last_change":"2024-03-12T02:54:07.251833+0000","last_active":"2024-03-12T02:54:31.006419+0000","last_peered":"2024-03-12T02:54:31.006419+0000","last_clean":"2024-03-12T02:54:31.006419+0000","last_became_active":"2024-03-12T02:54:07.250566+0000","last_became_peered":"2024-03-12T02:54:07.250566+0000","last_unstale":"2024-03-12T02:54:31.006419+0000","last_undegraded":"2024-03-12T02:54:31.006419+0000","last_fullsized":"2024-03-12T02:54:31.006419+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:01:20.257983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004338+0000","last_change":"2024-03-12T02:54:09.258514+0000","last_active":"2024-03-12T02:54:31.004338+0000","last_peered":"2024-03-12T02:54:31.004338+0000","last_clean":"2024-03-12T02:54:31.004338+0000","last_became_active":"2024-03-12T02:54:09.258199+0000","last_became_peered":"2024-03-12T02:54:09.258199+0000","last_unstale":"2024-03-12T02:54:31.004338+0000","last_undegraded":"2024-03-12T02:54:31.004338+0000","last_fullsized":"2024-03-12T02:54:31.004338+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:51:09.391155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004421+0000","last_change":"2024-03-12T02:54:03.222082+0000","last_active":"2024-03-12T02:54:31.004421+0000","last_peered":"2024-03-12T02:54:31.004421+0000","last_clean":"2024-03-12T02:54:31.004421+0000","last_became_active":"2024-03-12T02:54:03.221696+0000","last_became_peered":"2024-03-12T02:54:03.221696+0000","last_unstale":"2024-03-12T02:54:31.004421+0000","last_undegraded":"2024-03-12T02:54:31.004421+0000","last_fullsized":"2024-03-12T02:54:31.004421+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:17:18.708749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'25","reported_seq":71,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005136+0000","last_change":"2024-03-12T02:54:05.235363+0000","last_active":"2024-03-12T02:54:31.005136+0000","last_peered":"2024-03-12T02:54:31.005136+0000","last_clean":"2024-03-12T02:54:31.005136+0000","last_became_active":"2024-03-12T02:54:05.234994+0000","last_became_peered":"2024-03-12T02:54:05.234994+0000","last_unstale":"2024-03-12T02:54:31.005136+0000","last_undegraded":"2024-03-12T02:54:31.005136+0000","last_fullsized":"2024-03-12T02:54:31.005136+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:33:56.680419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003492+0000","last_change":"2024-03-12T02:54:07.256841+0000","last_active":"2024-03-12T02:54:31.003492+0000","last_peered":"2024-03-12T02:54:31.003492+0000","last_clean":"2024-03-12T02:54:31.003492+0000","last_became_active":"2024-03-12T02:54:07.256485+0000","last_became_peered":"2024-03-12T02:54:07.256485+0000","last_unstale":"2024-03-12T02:54:31.003492+0000","last_undegraded":"2024-03-12T02:54:31.003492+0000","last_fullsized":"2024-03-12T02:54:31.003492+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:52:10.673498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005395+0000","last_change":"2024-03-12T02:54:09.261808+0000","last_active":"2024-03-12T02:54:31.005395+0000","last_peered":"2024-03-12T02:54:31.005395+0000","last_clean":"2024-03-12T02:54:31.005395+0000","last_became_active":"2024-03-12T02:54:09.261452+0000","last_became_peered":"2024-03-12T02:54:09.261452+0000","last_unstale":"2024-03-12T02:54:31.005395+0000","last_undegraded":"2024-03-12T02:54:31.005395+0000","last_fullsized":"2024-03-12T02:54:31.005395+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:38:35.593312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'30","reported_seq":79,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006114+0000","last_change":"2024-03-12T02:54:05.234492+0000","last_active":"2024-03-12T02:54:31.006114+0000","last_peered":"2024-03-12T02:54:31.006114+0000","last_clean":"2024-03-12T02:54:31.006114+0000","last_became_active":"2024-03-12T02:54:05.234157+0000","last_became_peered":"2024-03-12T02:54:05.234157+0000","last_unstale":"2024-03-12T02:54:31.006114+0000","last_undegraded":"2024-03-12T02:54:31.006114+0000","last_fullsized":"2024-03-12T02:54:31.006114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:32:27.274904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002541+0000","last_change":"2024-03-12T02:54:03.227096+0000","last_active":"2024-03-12T02:54:31.002541+0000","last_peered":"2024-03-12T02:54:31.002541+0000","last_clean":"2024-03-12T02:54:31.002541+0000","last_became_active":"2024-03-12T02:54:03.226790+0000","last_became_peered":"2024-03-12T02:54:03.226790+0000","last_unstale":"2024-03-12T02:54:31.002541+0000","last_undegraded":"2024-03-12T02:54:31.002541+0000","last_fullsized":"2024-03-12T02:54:31.002541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:26:49.222298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005696+0000","last_change":"2024-03-12T02:54:07.259609+0000","last_active":"2024-03-12T02:54:31.005696+0000","last_peered":"2024-03-12T02:54:31.005696+0000","last_clean":"2024-03-12T02:54:31.005696+0000","last_became_active":"2024-03-12T02:54:07.259233+0000","last_became_peered":"2024-03-12T02:54:07.259233+0000","last_unstale":"2024-03-12T02:54:31.005696+0000","last_undegraded":"2024-03-12T02:54:31.005696+0000","last_fullsized":"2024-03-12T02:54:31.005696+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:31:04.355212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007075+0000","last_change":"2024-03-12T02:54:09.261527+0000","last_active":"2024-03-12T02:54:31.007075+0000","last_peered":"2024-03-12T02:54:31.007075+0000","last_clean":"2024-03-12T02:54:31.007075+0000","last_became_active":"2024-03-12T02:54:09.261128+0000","last_became_peered":"2024-03-12T02:54:09.261128+0000","last_unstale":"2024-03-12T02:54:31.007075+0000","last_undegraded":"2024-03-12T02:54:31.007075+0000","last_fullsized":"2024-03-12T02:54:31.007075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:50.566784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'16","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002632+0000","last_change":"2024-03-12T02:54:05.232529+0000","last_active":"2024-03-12T02:54:31.002632+0000","last_peered":"2024-03-12T02:54:31.002632+0000","last_clean":"2024-03-12T02:54:31.002632+0000","last_became_active":"2024-03-12T02:54:05.231838+0000","last_became_peered":"2024-03-12T02:54:05.231838+0000","last_unstale":"2024-03-12T02:54:31.002632+0000","last_undegraded":"2024-03-12T02:54:31.002632+0000","last_fullsized":"2024-03-12T02:54:31.002632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:25:51.846591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004903+0000","last_change":"2024-03-12T02:54:03.222035+0000","last_active":"2024-03-12T02:54:31.004903+0000","last_peered":"2024-03-12T02:54:31.004903+0000","last_clean":"2024-03-12T02:54:31.004903+0000","last_became_active":"2024-03-12T02:54:03.221065+0000","last_became_peered":"2024-03-12T02:54:03.221065+0000","last_unstale":"2024-03-12T02:54:31.004903+0000","last_undegraded":"2024-03-12T02:54:31.004903+0000","last_fullsized":"2024-03-12T02:54:31.004903+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:22:15.530498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:33.109182+0000","last_change":"2024-03-12T02:53:52.239954+0000","last_active":"2024-03-12T02:54:33.109182+0000","last_peered":"2024-03-12T02:54:33.109182+0000","last_clean":"2024-03-12T02:54:33.109182+0000","last_became_active":"2024-03-12T02:53:52.239662+0000","last_became_peered":"2024-03-12T02:53:52.239662+0000","last_unstale":"2024-03-12T02:54:33.109182+0000","last_undegraded":"2024-03-12T02:54:33.109182+0000","last_fullsized":"2024-03-12T02:54:33.109182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:53:51.228013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:53:51.228013+0000","last_clean_scrub_stamp":"2024-03-12T02:53:51.228013+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:40.204759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006379+0000","last_change":"2024-03-12T02:54:07.255847+0000","last_active":"2024-03-12T02:54:31.006379+0000","last_peered":"2024-03-12T02:54:31.006379+0000","last_clean":"2024-03-12T02:54:31.006379+0000","last_became_active":"2024-03-12T02:54:07.255500+0000","last_became_peered":"2024-03-12T02:54:07.255500+0000","last_unstale":"2024-03-12T02:54:31.006379+0000","last_undegraded":"2024-03-12T02:54:31.006379+0000","last_fullsized":"2024-03-12T02:54:31.006379+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:20.715553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005388+0000","last_change":"2024-03-12T02:54:09.257867+0000","last_active":"2024-03-12T02:54:31.005388+0000","last_peered":"2024-03-12T02:54:31.005388+0000","last_clean":"2024-03-12T02:54:31.005388+0000","last_became_active":"2024-03-12T02:54:09.257396+0000","last_became_peered":"2024-03-12T02:54:09.257396+0000","last_unstale":"2024-03-12T02:54:31.005388+0000","last_undegraded":"2024-03-12T02:54:31.005388+0000","last_fullsized":"2024-03-12T02:54:31.005388+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:20.260521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'20","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002574+0000","last_change":"2024-03-12T02:54:05.232455+0000","last_active":"2024-03-12T02:54:31.002574+0000","last_peered":"2024-03-12T02:54:31.002574+0000","last_clean":"2024-03-12T02:54:31.002574+0000","last_became_active":"2024-03-12T02:54:05.231898+0000","last_became_peered":"2024-03-12T02:54:05.231898+0000","last_unstale":"2024-03-12T02:54:31.002574+0000","last_undegraded":"2024-03-12T02:54:31.002574+0000","last_fullsized":"2024-03-12T02:54:31.002574+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:35:55.130489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003980+0000","last_change":"2024-03-12T02:54:03.227441+0000","last_active":"2024-03-12T02:54:31.003980+0000","last_peered":"2024-03-12T02:54:31.003980+0000","last_clean":"2024-03-12T02:54:31.003980+0000","last_became_active":"2024-03-12T02:54:03.227072+0000","last_became_peered":"2024-03-12T02:54:03.227072+0000","last_unstale":"2024-03-12T02:54:31.003980+0000","last_undegraded":"2024-03-12T02:54:31.003980+0000","last_fullsized":"2024-03-12T02:54:31.003980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:00:04.018362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008555+0000","last_change":"2024-03-12T02:54:07.256791+0000","last_active":"2024-03-12T02:54:31.008555+0000","last_peered":"2024-03-12T02:54:31.008555+0000","last_clean":"2024-03-12T02:54:31.008555+0000","last_became_active":"2024-03-12T02:54:07.256510+0000","last_became_peered":"2024-03-12T02:54:07.256510+0000","last_unstale":"2024-03-12T02:54:31.008555+0000","last_undegraded":"2024-03-12T02:54:31.008555+0000","last_fullsized":"2024-03-12T02:54:31.008555+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:59:13.677601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004256+0000","last_change":"2024-03-12T02:54:09.256439+0000","last_active":"2024-03-12T02:54:31.004256+0000","last_peered":"2024-03-12T02:54:31.004256+0000","last_clean":"2024-03-12T02:54:31.004256+0000","last_became_active":"2024-03-12T02:54:09.256076+0000","last_became_peered":"2024-03-12T02:54:09.256076+0000","last_unstale":"2024-03-12T02:54:31.004256+0000","last_undegraded":"2024-03-12T02:54:31.004256+0000","last_fullsized":"2024-03-12T02:54:31.004256+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:56:13.116283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008416+0000","last_change":"2024-03-12T02:54:03.223170+0000","last_active":"2024-03-12T02:54:31.008416+0000","last_peered":"2024-03-12T02:54:31.008416+0000","last_clean":"2024-03-12T02:54:31.008416+0000","last_became_active":"2024-03-12T02:54:03.222205+0000","last_became_peered":"2024-03-12T02:54:03.222205+0000","last_unstale":"2024-03-12T02:54:31.008416+0000","last_undegraded":"2024-03-12T02:54:31.008416+0000","last_fullsized":"2024-03-12T02:54:31.008416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:58.308197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'34","reported_seq":92,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:38.269603+0000","last_change":"2024-03-12T02:54:05.233098+0000","last_active":"2024-03-12T02:54:38.269603+0000","last_peered":"2024-03-12T02:54:38.269603+0000","last_clean":"2024-03-12T02:54:38.269603+0000","last_became_active":"2024-03-12T02:54:05.232761+0000","last_became_peered":"2024-03-12T02:54:05.232761+0000","last_unstale":"2024-03-12T02:54:38.269603+0000","last_undegraded":"2024-03-12T02:54:38.269603+0000","last_fullsized":"2024-03-12T02:54:38.269603+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:21:27.724795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:38.269743+0000","last_change":"2024-03-12T02:54:07.255825+0000","last_active":"2024-03-12T02:54:38.269743+0000","last_peered":"2024-03-12T02:54:38.269743+0000","last_clean":"2024-03-12T02:54:38.269743+0000","last_became_active":"2024-03-12T02:54:07.255519+0000","last_became_peered":"2024-03-12T02:54:07.255519+0000","last_unstale":"2024-03-12T02:54:38.269743+0000","last_undegraded":"2024-03-12T02:54:38.269743+0000","last_fullsized":"2024-03-12T02:54:38.269743+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:25:25.438343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003647+0000","last_change":"2024-03-12T02:54:09.263744+0000","last_active":"2024-03-12T02:54:31.003647+0000","last_peered":"2024-03-12T02:54:31.003647+0000","last_clean":"2024-03-12T02:54:31.003647+0000","last_became_active":"2024-03-12T02:54:09.263328+0000","last_became_peered":"2024-03-12T02:54:09.263328+0000","last_unstale":"2024-03-12T02:54:31.003647+0000","last_undegraded":"2024-03-12T02:54:31.003647+0000","last_fullsized":"2024-03-12T02:54:31.003647+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:35:28.529623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004927+0000","last_change":"2024-03-12T02:54:03.227074+0000","last_active":"2024-03-12T02:54:31.004927+0000","last_peered":"2024-03-12T02:54:31.004927+0000","last_clean":"2024-03-12T02:54:31.004927+0000","last_became_active":"2024-03-12T02:54:03.226718+0000","last_became_peered":"2024-03-12T02:54:03.226718+0000","last_unstale":"2024-03-12T02:54:31.004927+0000","last_undegraded":"2024-03-12T02:54:31.004927+0000","last_fullsized":"2024-03-12T02:54:31.004927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:29:39.215629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'19","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005937+0000","last_change":"2024-03-12T02:54:05.235241+0000","last_active":"2024-03-12T02:54:31.005937+0000","last_peered":"2024-03-12T02:54:31.005937+0000","last_clean":"2024-03-12T02:54:31.005937+0000","last_became_active":"2024-03-12T02:54:05.234850+0000","last_became_peered":"2024-03-12T02:54:05.234850+0000","last_unstale":"2024-03-12T02:54:31.005937+0000","last_undegraded":"2024-03-12T02:54:31.005937+0000","last_fullsized":"2024-03-12T02:54:31.005937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:08:33.940312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:38.269602+0000","last_change":"2024-03-12T02:54:07.257345+0000","last_active":"2024-03-12T02:54:38.269602+0000","last_peered":"2024-03-12T02:54:38.269602+0000","last_clean":"2024-03-12T02:54:38.269602+0000","last_became_active":"2024-03-12T02:54:07.256929+0000","last_became_peered":"2024-03-12T02:54:07.256929+0000","last_unstale":"2024-03-12T02:54:38.269602+0000","last_undegraded":"2024-03-12T02:54:38.269602+0000","last_fullsized":"2024-03-12T02:54:38.269602+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:17.720166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006987+0000","last_change":"2024-03-12T02:54:09.262431+0000","last_active":"2024-03-12T02:54:31.006987+0000","last_peered":"2024-03-12T02:54:31.006987+0000","last_clean":"2024-03-12T02:54:31.006987+0000","last_became_active":"2024-03-12T02:54:09.262197+0000","last_became_peered":"2024-03-12T02:54:09.262197+0000","last_unstale":"2024-03-12T02:54:31.006987+0000","last_undegraded":"2024-03-12T02:54:31.006987+0000","last_fullsized":"2024-03-12T02:54:31.006987+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:10:17.271515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006330+0000","last_change":"2024-03-12T02:54:03.228518+0000","last_active":"2024-03-12T02:54:31.006330+0000","last_peered":"2024-03-12T02:54:31.006330+0000","last_clean":"2024-03-12T02:54:31.006330+0000","last_became_active":"2024-03-12T02:54:03.228190+0000","last_became_peered":"2024-03-12T02:54:03.228190+0000","last_unstale":"2024-03-12T02:54:31.006330+0000","last_undegraded":"2024-03-12T02:54:31.006330+0000","last_fullsized":"2024-03-12T02:54:31.006330+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:32.314632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005154+0000","last_change":"2024-03-12T02:54:05.235485+0000","last_active":"2024-03-12T02:54:31.005154+0000","last_peered":"2024-03-12T02:54:31.005154+0000","last_clean":"2024-03-12T02:54:31.005154+0000","last_became_active":"2024-03-12T02:54:05.233642+0000","last_became_peered":"2024-03-12T02:54:05.233642+0000","last_unstale":"2024-03-12T02:54:31.005154+0000","last_undegraded":"2024-03-12T02:54:31.005154+0000","last_fullsized":"2024-03-12T02:54:31.005154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:17:25.837049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006306+0000","last_change":"2024-03-12T02:54:07.251600+0000","last_active":"2024-03-12T02:54:31.006306+0000","last_peered":"2024-03-12T02:54:31.006306+0000","last_clean":"2024-03-12T02:54:31.006306+0000","last_became_active":"2024-03-12T02:54:07.250757+0000","last_became_peered":"2024-03-12T02:54:07.250757+0000","last_unstale":"2024-03-12T02:54:31.006306+0000","last_undegraded":"2024-03-12T02:54:31.006306+0000","last_fullsized":"2024-03-12T02:54:31.006306+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:37:40.370418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005541+0000","last_change":"2024-03-12T02:54:09.257788+0000","last_active":"2024-03-12T02:54:31.005541+0000","last_peered":"2024-03-12T02:54:31.005541+0000","last_clean":"2024-03-12T02:54:31.005541+0000","last_became_active":"2024-03-12T02:54:09.257384+0000","last_became_peered":"2024-03-12T02:54:09.257384+0000","last_unstale":"2024-03-12T02:54:31.005541+0000","last_undegraded":"2024-03-12T02:54:31.005541+0000","last_fullsized":"2024-03-12T02:54:31.005541+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:39:51.047538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:38.269536+0000","last_change":"2024-03-12T02:54:07.256897+0000","last_active":"2024-03-12T02:54:38.269536+0000","last_peered":"2024-03-12T02:54:38.269536+0000","last_clean":"2024-03-12T02:54:38.269536+0000","last_became_active":"2024-03-12T02:54:07.256546+0000","last_became_peered":"2024-03-12T02:54:07.256546+0000","last_unstale":"2024-03-12T02:54:38.269536+0000","last_undegraded":"2024-03-12T02:54:38.269536+0000","last_fullsized":"2024-03-12T02:54:38.269536+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:07.972485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003775+0000","last_change":"2024-03-12T02:54:03.222122+0000","last_active":"2024-03-12T02:54:31.003775+0000","last_peered":"2024-03-12T02:54:31.003775+0000","last_clean":"2024-03-12T02:54:31.003775+0000","last_became_active":"2024-03-12T02:54:03.220839+0000","last_became_peered":"2024-03-12T02:54:03.220839+0000","last_unstale":"2024-03-12T02:54:31.003775+0000","last_undegraded":"2024-03-12T02:54:31.003775+0000","last_fullsized":"2024-03-12T02:54:31.003775+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:25.561859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'24","reported_seq":70,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006813+0000","last_change":"2024-03-12T02:54:05.234527+0000","last_active":"2024-03-12T02:54:31.006813+0000","last_peered":"2024-03-12T02:54:31.006813+0000","last_clean":"2024-03-12T02:54:31.006813+0000","last_became_active":"2024-03-12T02:54:05.234122+0000","last_became_peered":"2024-03-12T02:54:05.234122+0000","last_unstale":"2024-03-12T02:54:31.006813+0000","last_undegraded":"2024-03-12T02:54:31.006813+0000","last_fullsized":"2024-03-12T02:54:31.006813+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:31:33.618716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.009586+0000","last_change":"2024-03-12T02:54:09.265175+0000","last_active":"2024-03-12T02:54:31.009586+0000","last_peered":"2024-03-12T02:54:31.009586+0000","last_clean":"2024-03-12T02:54:31.009586+0000","last_became_active":"2024-03-12T02:54:09.264823+0000","last_became_peered":"2024-03-12T02:54:09.264823+0000","last_unstale":"2024-03-12T02:54:31.009586+0000","last_undegraded":"2024-03-12T02:54:31.009586+0000","last_fullsized":"2024-03-12T02:54:31.009586+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:47:36.980631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005562+0000","last_change":"2024-03-12T02:54:07.256584+0000","last_active":"2024-03-12T02:54:31.005562+0000","last_peered":"2024-03-12T02:54:31.005562+0000","last_clean":"2024-03-12T02:54:31.005562+0000","last_became_active":"2024-03-12T02:54:07.256147+0000","last_became_peered":"2024-03-12T02:54:07.256147+0000","last_unstale":"2024-03-12T02:54:31.005562+0000","last_undegraded":"2024-03-12T02:54:31.005562+0000","last_fullsized":"2024-03-12T02:54:31.005562+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:22:00.388166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002717+0000","last_change":"2024-03-12T02:54:03.227808+0000","last_active":"2024-03-12T02:54:31.002717+0000","last_peered":"2024-03-12T02:54:31.002717+0000","last_clean":"2024-03-12T02:54:31.002717+0000","last_became_active":"2024-03-12T02:54:03.227515+0000","last_became_peered":"2024-03-12T02:54:03.227515+0000","last_unstale":"2024-03-12T02:54:31.002717+0000","last_undegraded":"2024-03-12T02:54:31.002717+0000","last_fullsized":"2024-03-12T02:54:31.002717+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:01:19.152969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'27","reported_seq":72,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006588+0000","last_change":"2024-03-12T02:54:05.235996+0000","last_active":"2024-03-12T02:54:31.006588+0000","last_peered":"2024-03-12T02:54:31.006588+0000","last_clean":"2024-03-12T02:54:31.006588+0000","last_became_active":"2024-03-12T02:54:05.235736+0000","last_became_peered":"2024-03-12T02:54:05.235736+0000","last_unstale":"2024-03-12T02:54:31.006588+0000","last_undegraded":"2024-03-12T02:54:31.006588+0000","last_fullsized":"2024-03-12T02:54:31.006588+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:19:13.386593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007890+0000","last_change":"2024-03-12T02:54:09.262476+0000","last_active":"2024-03-12T02:54:31.007890+0000","last_peered":"2024-03-12T02:54:31.007890+0000","last_clean":"2024-03-12T02:54:31.007890+0000","last_became_active":"2024-03-12T02:54:09.262219+0000","last_became_peered":"2024-03-12T02:54:09.262219+0000","last_unstale":"2024-03-12T02:54:31.007890+0000","last_undegraded":"2024-03-12T02:54:31.007890+0000","last_fullsized":"2024-03-12T02:54:31.007890+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:37.785243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008062+0000","last_change":"2024-03-12T02:54:03.225015+0000","last_active":"2024-03-12T02:54:31.008062+0000","last_peered":"2024-03-12T02:54:31.008062+0000","last_clean":"2024-03-12T02:54:31.008062+0000","last_became_active":"2024-03-12T02:54:03.223307+0000","last_became_peered":"2024-03-12T02:54:03.223307+0000","last_unstale":"2024-03-12T02:54:31.008062+0000","last_undegraded":"2024-03-12T02:54:31.008062+0000","last_fullsized":"2024-03-12T02:54:31.008062+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:40:25.332902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'15","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002923+0000","last_change":"2024-03-12T02:54:05.232779+0000","last_active":"2024-03-12T02:54:31.002923+0000","last_peered":"2024-03-12T02:54:31.002923+0000","last_clean":"2024-03-12T02:54:31.002923+0000","last_became_active":"2024-03-12T02:54:05.231956+0000","last_became_peered":"2024-03-12T02:54:05.231956+0000","last_unstale":"2024-03-12T02:54:31.002923+0000","last_undegraded":"2024-03-12T02:54:31.002923+0000","last_fullsized":"2024-03-12T02:54:31.002923+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:21:26.413572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005105+0000","last_change":"2024-03-12T02:54:07.255192+0000","last_active":"2024-03-12T02:54:31.005105+0000","last_peered":"2024-03-12T02:54:31.005105+0000","last_clean":"2024-03-12T02:54:31.005105+0000","last_became_active":"2024-03-12T02:54:07.254615+0000","last_became_peered":"2024-03-12T02:54:07.254615+0000","last_unstale":"2024-03-12T02:54:31.005105+0000","last_undegraded":"2024-03-12T02:54:31.005105+0000","last_fullsized":"2024-03-12T02:54:31.005105+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:14:32.175047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003139+0000","last_change":"2024-03-12T02:54:09.261136+0000","last_active":"2024-03-12T02:54:31.003139+0000","last_peered":"2024-03-12T02:54:31.003139+0000","last_clean":"2024-03-12T02:54:31.003139+0000","last_became_active":"2024-03-12T02:54:09.260844+0000","last_became_peered":"2024-03-12T02:54:09.260844+0000","last_unstale":"2024-03-12T02:54:31.003139+0000","last_undegraded":"2024-03-12T02:54:31.003139+0000","last_fullsized":"2024-03-12T02:54:31.003139+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:53:17.706720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:38.269542+0000","last_change":"2024-03-12T02:54:07.256297+0000","last_active":"2024-03-12T02:54:38.269542+0000","last_peered":"2024-03-12T02:54:38.269542+0000","last_clean":"2024-03-12T02:54:38.269542+0000","last_became_active":"2024-03-12T02:54:07.255647+0000","last_became_peered":"2024-03-12T02:54:07.255647+0000","last_unstale":"2024-03-12T02:54:38.269542+0000","last_undegraded":"2024-03-12T02:54:38.269542+0000","last_fullsized":"2024-03-12T02:54:38.269542+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:40:13.586862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006870+0000","last_change":"2024-03-12T02:54:03.225714+0000","last_active":"2024-03-12T02:54:31.006870+0000","last_peered":"2024-03-12T02:54:31.006870+0000","last_clean":"2024-03-12T02:54:31.006870+0000","last_became_active":"2024-03-12T02:54:03.224262+0000","last_became_peered":"2024-03-12T02:54:03.224262+0000","last_unstale":"2024-03-12T02:54:31.006870+0000","last_undegraded":"2024-03-12T02:54:31.006870+0000","last_fullsized":"2024-03-12T02:54:31.006870+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:35:47.643035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'25","reported_seq":66,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.004521+0000","last_change":"2024-03-12T02:54:05.237794+0000","last_active":"2024-03-12T02:54:31.004521+0000","last_peered":"2024-03-12T02:54:31.004521+0000","last_clean":"2024-03-12T02:54:31.004521+0000","last_became_active":"2024-03-12T02:54:05.237263+0000","last_became_peered":"2024-03-12T02:54:05.237263+0000","last_unstale":"2024-03-12T02:54:31.004521+0000","last_undegraded":"2024-03-12T02:54:31.004521+0000","last_fullsized":"2024-03-12T02:54:31.004521+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:01.667418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006552+0000","last_change":"2024-03-12T02:54:09.260348+0000","last_active":"2024-03-12T02:54:31.006552+0000","last_peered":"2024-03-12T02:54:31.006552+0000","last_clean":"2024-03-12T02:54:31.006552+0000","last_became_active":"2024-03-12T02:54:09.259332+0000","last_became_peered":"2024-03-12T02:54:09.259332+0000","last_unstale":"2024-03-12T02:54:31.006552+0000","last_undegraded":"2024-03-12T02:54:31.006552+0000","last_fullsized":"2024-03-12T02:54:31.006552+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:58:04.309965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007977+0000","last_change":"2024-03-12T02:54:03.222629+0000","last_active":"2024-03-12T02:54:31.007977+0000","last_peered":"2024-03-12T02:54:31.007977+0000","last_clean":"2024-03-12T02:54:31.007977+0000","last_became_active":"2024-03-12T02:54:03.222111+0000","last_became_peered":"2024-03-12T02:54:03.222111+0000","last_unstale":"2024-03-12T02:54:31.007977+0000","last_undegraded":"2024-03-12T02:54:31.007977+0000","last_fullsized":"2024-03-12T02:54:31.007977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:50:58.162282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006625+0000","last_change":"2024-03-12T02:54:05.237864+0000","last_active":"2024-03-12T02:54:31.006625+0000","last_peered":"2024-03-12T02:54:31.006625+0000","last_clean":"2024-03-12T02:54:31.006625+0000","last_became_active":"2024-03-12T02:54:05.237519+0000","last_became_peered":"2024-03-12T02:54:05.237519+0000","last_unstale":"2024-03-12T02:54:31.006625+0000","last_undegraded":"2024-03-12T02:54:31.006625+0000","last_fullsized":"2024-03-12T02:54:31.006625+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:27:54.863526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003998+0000","last_change":"2024-03-12T02:54:07.256148+0000","last_active":"2024-03-12T02:54:31.003998+0000","last_peered":"2024-03-12T02:54:31.003998+0000","last_clean":"2024-03-12T02:54:31.003998+0000","last_became_active":"2024-03-12T02:54:07.255589+0000","last_became_peered":"2024-03-12T02:54:07.255589+0000","last_unstale":"2024-03-12T02:54:31.003998+0000","last_undegraded":"2024-03-12T02:54:31.003998+0000","last_fullsized":"2024-03-12T02:54:31.003998+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:47:32.266619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.001965+0000","last_change":"2024-03-12T02:54:09.258391+0000","last_active":"2024-03-12T02:54:31.001965+0000","last_peered":"2024-03-12T02:54:31.001965+0000","last_clean":"2024-03-12T02:54:31.001965+0000","last_became_active":"2024-03-12T02:54:09.258041+0000","last_became_peered":"2024-03-12T02:54:09.258041+0000","last_unstale":"2024-03-12T02:54:31.001965+0000","last_undegraded":"2024-03-12T02:54:31.001965+0000","last_fullsized":"2024-03-12T02:54:31.001965+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:25:17.346673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.008273+0000","last_change":"2024-03-12T02:54:03.224206+0000","last_active":"2024-03-12T02:54:31.008273+0000","last_peered":"2024-03-12T02:54:31.008273+0000","last_clean":"2024-03-12T02:54:31.008273+0000","last_became_active":"2024-03-12T02:54:03.223280+0000","last_became_peered":"2024-03-12T02:54:03.223280+0000","last_unstale":"2024-03-12T02:54:31.008273+0000","last_undegraded":"2024-03-12T02:54:31.008273+0000","last_fullsized":"2024-03-12T02:54:31.008273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:24.500603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005389+0000","last_change":"2024-03-12T02:54:05.238220+0000","last_active":"2024-03-12T02:54:31.005389+0000","last_peered":"2024-03-12T02:54:31.005389+0000","last_clean":"2024-03-12T02:54:31.005389+0000","last_became_active":"2024-03-12T02:54:05.237884+0000","last_became_peered":"2024-03-12T02:54:05.237884+0000","last_unstale":"2024-03-12T02:54:31.005389+0000","last_undegraded":"2024-03-12T02:54:31.005389+0000","last_fullsized":"2024-03-12T02:54:31.005389+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:17:41.716484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.002459+0000","last_change":"2024-03-12T02:54:07.254060+0000","last_active":"2024-03-12T02:54:31.002459+0000","last_peered":"2024-03-12T02:54:31.002459+0000","last_clean":"2024-03-12T02:54:31.002459+0000","last_became_active":"2024-03-12T02:54:07.253067+0000","last_became_peered":"2024-03-12T02:54:07.253067+0000","last_unstale":"2024-03-12T02:54:31.002459+0000","last_undegraded":"2024-03-12T02:54:31.002459+0000","last_fullsized":"2024-03-12T02:54:31.002459+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:05.305908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.003772+0000","last_change":"2024-03-12T02:54:09.263618+0000","last_active":"2024-03-12T02:54:31.003772+0000","last_peered":"2024-03-12T02:54:31.003772+0000","last_clean":"2024-03-12T02:54:31.003772+0000","last_became_active":"2024-03-12T02:54:09.262535+0000","last_became_peered":"2024-03-12T02:54:09.262535+0000","last_unstale":"2024-03-12T02:54:31.003772+0000","last_undegraded":"2024-03-12T02:54:31.003772+0000","last_fullsized":"2024-03-12T02:54:31.003772+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:08.242835+0000","last_clean_scrub_stamp":"2024-03-12T02:54:08.242835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:55:18.615404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.005061+0000","last_change":"2024-03-12T02:54:03.227748+0000","last_active":"2024-03-12T02:54:31.005061+0000","last_peered":"2024-03-12T02:54:31.005061+0000","last_clean":"2024-03-12T02:54:31.005061+0000","last_became_active":"2024-03-12T02:54:03.227486+0000","last_became_peered":"2024-03-12T02:54:03.227486+0000","last_unstale":"2024-03-12T02:54:31.005061+0000","last_undegraded":"2024-03-12T02:54:31.005061+0000","last_fullsized":"2024-03-12T02:54:31.005061+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:02.201468+0000","last_clean_scrub_stamp":"2024-03-12T02:54:02.201468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:49:53.193904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.006490+0000","last_change":"2024-03-12T02:54:05.238501+0000","last_active":"2024-03-12T02:54:31.006490+0000","last_peered":"2024-03-12T02:54:31.006490+0000","last_clean":"2024-03-12T02:54:31.006490+0000","last_became_active":"2024-03-12T02:54:05.238084+0000","last_became_peered":"2024-03-12T02:54:05.238084+0000","last_unstale":"2024-03-12T02:54:31.006490+0000","last_undegraded":"2024-03-12T02:54:31.006490+0000","last_fullsized":"2024-03-12T02:54:31.006490+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:04.220460+0000","last_clean_scrub_stamp":"2024-03-12T02:54:04.220460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:05:02.591065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-12T02:54:31.007210+0000","last_change":"2024-03-12T02:54:07.256290+0000","last_active":"2024-03-12T02:54:31.007210+0000","last_peered":"2024-03-12T02:54:31.007210+0000","last_clean":"2024-03-12T02:54:31.007210+0000","last_became_active":"2024-03-12T02:54:07.256011+0000","last_became_peered":"2024-03-12T02:54:07.256011+0000","last_unstale":"2024-03-12T02:54:31.007210+0000","last_undegraded":"2024-03-12T02:54:31.007210+0000","last_fullsized":"2024-03-12T02:54:31.007210+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T02:54:06.230651+0000","last_clean_scrub_stamp":"2024-03-12T02:54:06.230651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:54:05.024023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":588,"ondisk_log_size":588,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T02:54:48.906 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-12T02:54:48.906 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-12T02:54:48.906 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-12T02:54:48.907 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-12T02:54:50.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:50 smithi079 ceph-mon[25794]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:50.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:50 smithi115 ceph-mon[29819]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:50.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:50 smithi129 ceph-mon[30321]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:51.019 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:50 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ::ffff:172.21.15.115 - - [12/Mar/2024:02:54:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T02:54:51.116 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:51.649 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:51 smithi079 systemd[1]: Starting Ceph alertmanager.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:54:52.268 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:52 smithi079 ceph-mon[25794]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:52.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:52 smithi115 ceph-mon[29819]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:52.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:52 smithi129 ceph-mon[30321]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:52.981 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:52 smithi079 podman[53312]: 2024-03-12 02:54:52.670454059 +0000 UTC m=+0.104404959 volume create 498ce7e34d941e2b48fecbda161543519d83d51e0b3e1a39d031cda646bac167 2024-03-12T02:54:52.981 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:52 smithi079 podman[53312]: 2024-03-12 02:54:52.585537561 +0000 UTC m=+0.019488495 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-12T02:54:52.981 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:52 smithi079 podman[53312]: 2024-03-12 02:54:52.762419342 +0000 UTC m=+0.196370283 container create 03a2b1dc0bb16ba2a28be486f1c371da8e6007c9e067f6d675923f91fbc906c6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T02:54:53.406 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 podman[53312]: 2024-03-12 02:54:53.130709988 +0000 UTC m=+0.564660899 container init 03a2b1dc0bb16ba2a28be486f1c371da8e6007c9e067f6d675923f91fbc906c6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 podman[53312]: 2024-03-12 02:54:53.136198024 +0000 UTC m=+0.570148926 container start 03a2b1dc0bb16ba2a28be486f1c371da8e6007c9e067f6d675923f91fbc906c6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:54:53.148Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:54:53.148Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:54:53.149Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.79 port=9094 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:54:53.150Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:54:53.174Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:54:53.175Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:54:53.177Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:54:53.177Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 bash[53312]: 03a2b1dc0bb16ba2a28be486f1c371da8e6007c9e067f6d675923f91fbc906c6 2024-03-12T02:54:53.407 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:53 smithi079 systemd[1]: Started Ceph alertmanager.a for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:54:53.716 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-12T02:54:53.717 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-12T02:54:53.717 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:54:53] ENGINE Bus STOPPING 2024-03-12T02:54:53.717 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:54:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T02:54:53.717 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:54:53] ENGINE Bus STOPPED 2024-03-12T02:54:53.717 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:54:53] ENGINE Bus STARTING 2024-03-12T02:54:53.717 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:54:53] ENGINE Serving on http://:::9283 2024-03-12T02:54:53.717 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:54:53 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:54:53] ENGINE Bus STARTED 2024-03-12T02:54:54.317 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-12T02:54:54.317 INFO:tasks.cephadm:Setup complete, yielding 2024-03-12T02:54:54.318 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T02:54:54.327 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-03-12T02:54:54.327 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-12T02:54:54.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:54.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T02:54:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T02:54:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T02:54:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: Deploying daemon grafana.a on smithi115 2024-03-12T02:54:54.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:54 smithi079 ceph-mon[25794]: from='client.? 172.21.15.79:0/319875273' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T02:54:54.578 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:54.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:54.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T02:54:54.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T02:54:54.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T02:54:54.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: Deploying daemon grafana.a on smithi115 2024-03-12T02:54:54.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:54 smithi115 ceph-mon[29819]: from='client.? 172.21.15.79:0/319875273' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T02:54:54.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:54.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T02:54:54.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T02:54:54.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T02:54:54.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:54.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: Deploying daemon grafana.a on smithi115 2024-03-12T02:54:54.629 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:54 smithi129 ceph-mon[30321]: from='client.? 172.21.15.79:0/319875273' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T02:54:55.455 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:54:55 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:54:55.151Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001049388s 2024-03-12T02:54:56.240 INFO:teuthology.orchestra.run.smithi079.stdout:167 167 2024-03-12T02:54:56.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:56 smithi079 ceph-mon[25794]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:56.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:56 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:56.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:56 smithi115 ceph-mon[29819]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:56.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:56 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:56.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:56 smithi129 ceph-mon[30321]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:56.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:56 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:54:56.871 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-12T02:54:57.107 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:54:58.501 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:58 smithi079 ceph-mon[25794]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:58.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:58 smithi115 ceph-mon[29819]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:58.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:58 smithi129 ceph-mon[30321]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:54:58.923 INFO:teuthology.orchestra.run.smithi079.stdout:Backend: cephadm 2024-03-12T02:54:58.923 INFO:teuthology.orchestra.run.smithi079.stdout:Available: Yes 2024-03-12T02:54:58.923 INFO:teuthology.orchestra.run.smithi079.stdout:Paused: No 2024-03-12T02:54:59.557 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:54:59 smithi079 ceph-mon[25794]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:59.586 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-12T02:54:59.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:54:59 smithi115 ceph-mon[29819]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:59.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:54:59 smithi129 ceph-mon[30321]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:54:59.820 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:55:00.560 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:00 smithi079 ceph-mon[25794]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:00.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:00 smithi115 ceph-mon[29819]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:00.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:00 smithi129 ceph-mon[30321]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:00.938 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:00 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ::ffff:172.21.15.115 - - [12/Mar/2024:02:55:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T02:55:01.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:01 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:55:01.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:01 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:55:01.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:01 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:55:01.661 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T02:55:01.661 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.a smithi079 *:9093,9094 starting - - - - 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.x smithi115 *:8443,8765 running (2m) 26s ago 2m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 43d1f31507b4 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.y smithi079 *:9283,8765 running (4m) 23s ago 4m 481M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 13cc77d2a103 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.z smithi129 *:8443,8765 running (2m) 28s ago 2m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bbfacb7d3ec9 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:mon.a smithi079 running (4m) 23s ago 4m 49.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 527ba2953c6b 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:mon.b smithi115 running (2m) 26s ago 2m 41.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3e348d0eae3a 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:mon.c smithi129 running (2m) 28s ago 2m 46.0M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c148043ea86 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (2m) 23s ago 2m 51.1M 15.6G 19.0.0-1979-g2b0101eb 4133f8bc41bb cc6516580864 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi115 running (101s) 26s ago 101s 60.8M 15.6G 19.0.0-1979-g2b0101eb 4133f8bc41bb 9128a0afb07d 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi129 running (78s) 28s ago 77s 50.8M 16.6G 19.0.0-1979-g2b0101eb 4133f8bc41bb 795804c2be19 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.a smithi115 *:9095 running (39s) 26s ago 38s 25.1M - 2.43.0 a07b618ecd1d 85e505665ae8 2024-03-12T02:55:01.662 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.foo.a smithi079 *:80 running (60s) 23s ago 60s 77.5M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 7d29761e6c9d 2024-03-12T02:55:02.323 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-12T02:55:02.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:02 smithi079 ceph-mon[25794]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:02.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:02 smithi079 ceph-mon[25794]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:02.561 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:55:02.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:02 smithi115 ceph-mon[29819]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:02.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:02 smithi115 ceph-mon[29819]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:02.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:02 smithi129 ceph-mon[30321]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:02.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:02 smithi129 ceph-mon[30321]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:03.309 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:55:03 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:55:03.153Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003289424s 2024-03-12T02:55:04.398 INFO:teuthology.orchestra.run.smithi079.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-12T02:55:04.398 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager ?:9093,9094 0/1 - 58s smithi079=a;count:1 2024-03-12T02:55:04.398 INFO:teuthology.orchestra.run.smithi079.stdout:grafana ?:3000 0/1 - 55s smithi115=a;count:1 2024-03-12T02:55:04.398 INFO:teuthology.orchestra.run.smithi079.stdout:mgr 3/3 31s ago 2m smithi079=y;smithi115=x;smithi129=z;count:3 2024-03-12T02:55:04.398 INFO:teuthology.orchestra.run.smithi079.stdout:mon 3/3 31s ago 2m smithi079:172.21.15.79=a;smithi115:172.21.15.115=b;smithi129:172.21.15.129=c;count:3 2024-03-12T02:55:04.399 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter ?:9100 0/3 - 61s smithi079=a;smithi115=b;smithi129=c;count:3 2024-03-12T02:55:04.399 INFO:teuthology.orchestra.run.smithi079.stdout:osd 3 31s ago - 2024-03-12T02:55:04.399 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus ?:9095 1/1 29s ago 63s smithi115=a;count:1 2024-03-12T02:55:04.399 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.foo.a ?:80 1/1 26s ago 62s smithi079=foo.a;count:1 2024-03-12T02:55:04.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:04 smithi115 ceph-mon[29819]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:04.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:04 smithi129 ceph-mon[30321]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:04.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:04 smithi079 ceph-mon[25794]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:05.034 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-12T02:55:05.269 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:55:05.568 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:05 smithi079 ceph-mon[25794]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:05.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:05 smithi115 ceph-mon[29819]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:05.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:05 smithi129 ceph-mon[30321]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:06.568 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:06 smithi079 ceph-mon[25794]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:06.574 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:06 smithi115 ceph-mon[29819]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:06.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:06 smithi129 ceph-mon[30321]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:06.984 INFO:teuthology.orchestra.run.smithi079.stdout:HOST ADDR LABELS STATUS 2024-03-12T02:55:06.984 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 172.21.15.79 2024-03-12T02:55:06.984 INFO:teuthology.orchestra.run.smithi079.stdout:smithi115 172.21.15.115 2024-03-12T02:55:06.985 INFO:teuthology.orchestra.run.smithi079.stdout:smithi129 172.21.15.129 2024-03-12T02:55:06.985 INFO:teuthology.orchestra.run.smithi079.stdout:3 hosts in cluster 2024-03-12T02:55:07.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-12T02:55:07.855 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:55:08.320 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 systemd[1]: Starting Ceph grafana.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:08.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:08 smithi079 ceph-mon[25794]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:08.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:08 smithi079 ceph-mon[25794]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:08.589 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-mon[29819]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:08.590 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-mon[29819]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:08.590 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 podman[42629]: 2024-03-12 02:55:08.308965332 +0000 UTC m=+0.017610944 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-12T02:55:08.590 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 podman[42629]: 2024-03-12 02:55:08.431033908 +0000 UTC m=+0.139679521 container create 776fa070fbb21c274e66ab9bce5202cbda5f6d5ad32526a0e7b1cafe18c5e319 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, name=ubi8, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, maintainer=Paul Cuzner , distribution-scope=public, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5) 2024-03-12T02:55:08.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:08 smithi129 ceph-mon[30321]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:08.628 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:08 smithi129 ceph-mon[30321]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:08.849 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 podman[42629]: 2024-03-12 02:55:08.700194941 +0000 UTC m=+0.408840566 container init 776fa070fbb21c274e66ab9bce5202cbda5f6d5ad32526a0e7b1cafe18c5e319 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, version=8.5, release=239.1651231664, architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, distribution-scope=public, vcs-type=git) 2024-03-12T02:55:08.850 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 podman[42629]: 2024-03-12 02:55:08.705429049 +0000 UTC m=+0.414074662 container start 776fa070fbb21c274e66ab9bce5202cbda5f6d5ad32526a0e7b1cafe18c5e319 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, name=ubi8, io.buildah.version=1.32.0, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, vendor=Red Hat, Inc.) 2024-03-12T02:55:08.850 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 bash[42629]: 776fa070fbb21c274e66ab9bce5202cbda5f6d5ad32526a0e7b1cafe18c5e319 2024-03-12T02:55:08.850 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 systemd[1]: Started Ceph grafana.a for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:55:09.113 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.859799946Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-12T02:55:09.113 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860021493Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-12T02:55:09.113 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860061401Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-12T02:55:09.113 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860073428Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860082842Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860089442Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860095327Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860101239Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860107161Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860114399Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860119954Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860125359Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860131617Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=settings t=2024-03-12T02:55:08.860137468Z level=info msg="App mode production" 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=sqlstore t=2024-03-12T02:55:08.860188168Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-12T02:55:09.114 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=sqlstore t=2024-03-12T02:55:08.860205914Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.86106492Z level=info msg="Starting DB migrations" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.861143117Z level=info msg="Executing migration" id="create migration_log table" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.861645189Z level=info msg="Executing migration" id="create user table" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.862099373Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.86261295Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.863063974Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.863650293Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.864070508Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.864819346Z level=info msg="Executing migration" id="create user table v2" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.865264244Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.865711187Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.866154721Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.866525944Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-12T02:55:09.115 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.866872427Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-12T02:55:09.116 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.867323868Z level=info msg="Executing migration" id="Update user table charset" 2024-03-12T02:55:09.116 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.86765032Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-12T02:55:09.116 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.868077305Z level=info msg="Executing migration" id="Add missing user data" 2024-03-12T02:55:09.116 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.868453839Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-12T02:55:09.116 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.868865218Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-12T02:55:09.116 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.869294235Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-12T02:55:09.116 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.869734262Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-12T02:55:09.116 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.87087495Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-12T02:55:09.117 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.871360455Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.871804497Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.872217259Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.873081927Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.873912352Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.874738665Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.875598881Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.876499362Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.877383247Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.878212543Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.879076895Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.879908249Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.880798353Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-12T02:55:09.118 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.881649127Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-12T02:55:09.119 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.882554666Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-12T02:55:09.119 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.88341521Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-12T02:55:09.119 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.886227685Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-12T02:55:09.119 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.888942397Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-12T02:55:09.119 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.891557813Z level=info msg="Executing migration" id="create star table" 2024-03-12T02:55:09.119 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.891949015Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-12T02:55:09.119 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.894668101Z level=info msg="Executing migration" id="create org table v1" 2024-03-12T02:55:09.119 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.897368948Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-12T02:55:09.119 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.900086657Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-12T02:55:09.119 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.902863376Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.905625176Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.906043426Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.908433518Z level=info msg="Executing migration" id="Update org table charset" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.911195464Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.91387132Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.916583186Z level=info msg="Executing migration" id="create dashboard table" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.916976972Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.919732565Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.92249677Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.925141954Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.925560223Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.928143797Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-12T02:55:09.120 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.928995338Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.930894624Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.933441868Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.933893255Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.936531962Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.939129774Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.941831561Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.944405065Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.947131111Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.949826487Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.95234627Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.952846964Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.955280018Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.958020691Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-12T02:55:09.121 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.960627224Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.960947591Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.961762092Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.962272852Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.96310806Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.963615953Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.964437267Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.964899857Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.965316068Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.966189082Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.966637336Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.96750286Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.968408607Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.969264975Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-12T02:55:09.122 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.969697819Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.970499568Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.971322803Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.972170069Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.972685123Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.97351013Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.973857403Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.974354797Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.975241068Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.975791293Z level=info msg="Executing migration" id="create data_source table" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.976656062Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.977461232Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.977903096Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.978371205Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-12T02:55:09.123 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.979244154Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.980237781Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.981112335Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.981531904Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.982396622Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.983253557Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.984106622Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.984673643Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.984950435Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.985484877Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.986015434Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.986349258Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.986636066Z level=info msg="Executing migration" id="Add uid column" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.987178352Z level=info msg="Executing migration" id="Update uid value" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.987990461Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-12T02:55:09.124 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.988448002Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-12T02:55:09.125 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.99128447Z level=info msg="Executing migration" id="create api_key table" 2024-03-12T02:55:09.125 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.991697262Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-12T02:55:09.125 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.994339755Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-12T02:55:09.125 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.997063608Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-12T02:55:09.125 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:08 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:08.999891124Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-12T02:55:09.125 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.000325291Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-12T02:55:09.125 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.000723194Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-12T02:55:09.125 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.00110074Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.002120698Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.0025179Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.002893685Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.003311322Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.00372478Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.004032223Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.004400194Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.006592756Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.007167531Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.009328375Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.010178997Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-12T02:55:09.126 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.011044934Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.011890605Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.012704893Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.013571995Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.01441447Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.01527966Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.01613465Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.016944808Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.017827742Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.018681149Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.019546707Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.020395035Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-12T02:55:09.127 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.021245817Z level=info msg="Executing migration" id="create quota table v1" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.022100327Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.022954687Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.02381581Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.024671106Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.025544842Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.026422248Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.027251531Z level=info msg="Executing migration" id="create session table" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.028147805Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.029081988Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.029938615Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.030833296Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.03172266Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.032515758Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-12T02:55:09.128 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.033406343Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.034287516Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.035141426Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.036013326Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.036892849Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.037725865Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.038622395Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.041388747Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.044189086Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.04714414Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.050069299Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.05293473Z level=info msg="Executing migration" id="create alert table v1" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.055729569Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.058578926Z level=info msg="Executing migration" id="add index alert state" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.061503Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-12T02:55:09.129 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.064321059Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-12T02:55:09.130 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.06715283Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-12T02:55:09.130 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.070123925Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-12T02:55:09.130 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.072914206Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-12T02:55:09.130 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.075758594Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-12T02:55:09.130 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.078499435Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-12T02:55:09.130 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.081379821Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-12T02:55:09.130 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.084373301Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-12T02:55:09.130 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.087170299Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-12T02:55:09.130 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.089908889Z level=info msg="Executing migration" id="Add column is_default" 2024-03-12T02:55:09.131 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.092699831Z level=info msg="Executing migration" id="Add column frequency" 2024-03-12T02:55:09.131 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.095249052Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-12T02:55:09.131 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.097890579Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-12T02:55:09.131 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.100638538Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-12T02:55:09.131 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.103418638Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-12T02:55:09.131 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.106133338Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-12T02:55:09.131 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.108711887Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-12T02:55:09.131 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.111270007Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-12T02:55:09.131 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.114055814Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-12T02:55:09.131 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.11666409Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-12T02:55:09.366 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.119343722Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-12T02:55:09.366 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.1220342Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-12T02:55:09.366 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.124730774Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-12T02:55:09.366 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.127436216Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-12T02:55:09.366 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.130178451Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-12T02:55:09.366 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.132601984Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-12T02:55:09.366 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.13528292Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-12T02:55:09.366 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.136118549Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-12T02:55:09.366 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.13699416Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-12T02:55:09.366 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.137849739Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.138700528Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.139582355Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.140422259Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.141269438Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.14216896Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.143006489Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.143818069Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.144705876Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.145523102Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.146380924Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.147233425Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.148071234Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.148955599Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-12T02:55:09.367 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.149816802Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.150660491Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.152238155Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.153109497Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.153961006Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.15481061Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.155694813Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.156545378Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.157475698Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.15832716Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.159176674Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.160033258Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.16091863Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-12T02:55:09.368 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.161804357Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.16265025Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.163498945Z level=info msg="Executing migration" id="Move region to single row" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.164372528Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.165245941Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.166120244Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.168936515Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.171932501Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.17479354Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.177680946Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.18047024Z level=info msg="Executing migration" id="create test_data table" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.183440574Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-12T02:55:09.370 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.186371529Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.189269676Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.192149515Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.195031731Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.197834407Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.200784859Z level=info msg="Executing migration" id="create team table" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.20349444Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.206241244Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.208966094Z level=info msg="Executing migration" id="create team member table" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.211766185Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.214492821Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.217430356Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.220061982Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.222673917Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-12T02:55:09.371 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.225337306Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.227932698Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.230679264Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.231624286Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.234373613Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.234878139Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.237866074Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.243252123Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.243688584Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.244102281Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.244537277Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.244803135Z level=info msg="Executing migration" id="create tag table" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.245187069Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.245597451Z level=info msg="Executing migration" id="create login attempt table" 2024-03-12T02:55:09.372 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.245976036Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.246585912Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.246956799Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.248841525Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.249234329Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.249649426Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.249933377Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.250299939Z level=info msg="Executing migration" id="create user auth table" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.250669057Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.251076574Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.251365848Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.252075096Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.252792551Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.253562219Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.254309405Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.254725747Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-12T02:55:09.373 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.255491656Z level=info msg="Executing migration" id="create server_lock table" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.255874764Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.256326909Z level=info msg="Executing migration" id="create user auth token table" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.256738112Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.257143692Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.257549079Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.257945985Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.258693367Z level=info msg="Executing migration" id="create cache_data table" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.259094009Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.259510582Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.259916354Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.260355177Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.260640657Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.260942733Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.262655596Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-12T02:55:09.374 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.265449271Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.268102536Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.26839477Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.268772502Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.269209589Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.271540095Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.274239784Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.275021844Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.277820518Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.280285124Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.282934639Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.283408759Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.286008382Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.28870691Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.291419622Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-12T02:55:09.375 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.291845205Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.294543265Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.297180662Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.299815902Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.300261359Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.303006197Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.306406197Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.309762081Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.310278677Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.311121591Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.311945453Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.312402754Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.312851789Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.313717491Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.314545555Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.315384115Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-12T02:55:09.376 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.316213959Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.317073082Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.31792698Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.318776383Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.31923439Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.320082338Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.320932152Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.321789388Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.322673792Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.323532369Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.324346281Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.325213855Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.32569155Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.326509012Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-12T02:55:09.377 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.326836586Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.327747497Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.328641774Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.329542574Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.330425404Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.331269444Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.33158125Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.332500311Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.333348668Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.334193416Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.335112539Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.335991927Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.336892826Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-12T02:55:09.378 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.337772332Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.338245499Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.341040818Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.343765604Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.346462172Z level=info msg="Executing migration" id="create alert_image table" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.346880905Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.349682457Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.352335879Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.352819589Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.355122963Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.355214203Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.357925335Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.360688011Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.363719767Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.36388785Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-12T02:55:09.379 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.364057382Z level=info msg="alerts found to migrate" alerts=0 2024-03-12T02:55:09.380 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.364526748Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-12T02:55:09.630 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.369096544Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-12T02:55:09.630 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.371775307Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-12T02:55:09.630 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.374548806Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-12T02:55:09.630 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.375071709Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-12T02:55:09.630 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.377793383Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.380492636Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.383389979Z level=info msg="Executing migration" id="create data_keys table" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.383828169Z level=info msg="Executing migration" id="create secrets table" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.386552601Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.390886877Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.393410386Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.393726482Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.397938779Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.402138814Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.402541922Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.404784699Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.40515278Z level=info msg="Executing migration" id="create permission table" 2024-03-12T02:55:09.631 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.405545587Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-12T02:55:09.632 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.405955648Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.406393201Z level=info msg="Executing migration" id="create role table" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.409261979Z level=info msg="Executing migration" id="add column display_name" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.411841691Z level=info msg="Executing migration" id="add column group_name" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.414286677Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.414739273Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.417461797Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.420303863Z level=info msg="Executing migration" id="create team role table" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.422819373Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.423304847Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.426053924Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.428818396Z level=info msg="Executing migration" id="create user role table" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.431231443Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-12T02:55:09.633 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.431667415Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.434350182Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.437214301Z level=info msg="Executing migration" id="create builtin role table" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.439846137Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.440497196Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.44293039Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.445574923Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.44641935Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.44686508Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.447849882Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.448579117Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.449539892Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.45012918Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.451093053Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-12T02:55:09.634 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.452118335Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.452795945Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.453091108Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.453542026Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.4539531Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.454364574Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.454838715Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.455669229Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.45613996Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.456973817Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.457977551Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.459001571Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.459594678Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.460609872Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.46136924Z level=info msg="Executing migration" id="add correlation config column" 2024-03-12T02:55:09.635 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.462468019Z level=info msg="Executing migration" id="create entity_events table" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.46297585Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.463838903Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.463931604Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.464670125Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.464756949Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.465505133Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.465974892Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.466843667Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.467720567Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.468559562Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.469044023Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.470043389Z level=info msg="Executing migration" id="Drop public config table" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.470798936Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.471738901Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-12T02:55:09.636 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.472403309Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.475435764Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.478339215Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.481508368Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.482560216Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.485573982Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.488439257Z level=info msg="Executing migration" id="add share column" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.491290412Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.49412838Z level=info msg="Executing migration" id="create file table" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.497009258Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.499831365Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.50246653Z level=info msg="Executing migration" id="create file_meta table" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.502883058Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.505627549Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.50829461Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.511240941Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-12T02:55:09.637 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.514018386Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.516938653Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.519711156Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.52219819Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.524968059Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.527479696Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.530294267Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.532981489Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.535656919Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.538517345Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.541362148Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.551173908Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.553754151Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-12T02:55:09.638 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.556527068Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-12T02:55:09.639 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.560054537Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-12T02:55:09.639 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.562841767Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-12T02:55:09.639 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.565620497Z level=info msg="Executing migration" id="create folder table" 2024-03-12T02:55:09.639 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.568153989Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-12T02:55:09.639 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.570889514Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-12T02:55:09.639 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.57171989Z level=info msg="Executing migration" id="Update folder title length" 2024-03-12T02:55:09.639 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.572467162Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-12T02:55:09.639 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=migrator t=2024-03-12T02:55:09.573300324Z level=info msg="migrations completed" performed=485 skipped=0 duration=712.181725ms 2024-03-12T02:55:09.639 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=sqlstore t=2024-03-12T02:55:09.573687632Z level=info msg="Created default organization" 2024-03-12T02:55:09.639 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=plugin.loader t=2024-03-12T02:55:09.628245366Z level=info msg="Plugin registered" pluginID=input 2024-03-12T02:55:09.690 INFO:teuthology.orchestra.run.smithi079.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-12T02:55:09.690 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013G400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-12T02:55:09.690 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 /dev/nvme1n1 ssd Linux_1d4d75b9f456e2261dcf 89.4G Yes 2m ago 2024-03-12T02:55:09.690 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 /dev/nvme2n1 ssd Linux_24de07118c241a1889d7 89.4G Yes 2m ago 2024-03-12T02:55:09.690 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 /dev/nvme3n1 ssd Linux_8df46f27f165619ab8c3 89.4G Yes 2m ago 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 /dev/nvme4n1 ssd Linux_b7f4da26637cfca1a901 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi115 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UU400BGN 372G No 102s ago Has a FileSystem, LVM detected 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi115 /dev/nvme1n1 ssd Linux_bcd78b3fdfcd2cc84856 89.4G Yes 102s ago 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi115 /dev/nvme2n1 ssd Linux_0fc1458256c92e106253 89.4G Yes 102s ago 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi115 /dev/nvme3n1 ssd Linux_c71e70068515d2388ab9 89.4G Yes 102s ago 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi115 /dev/nvme4n1 ssd Linux_4a5c9d16d79f34b54f07 89.4G No 102s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi129 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300HY400BGN 372G No 78s ago Has a FileSystem, LVM detected 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi129 /dev/nvme1n1 ssd Linux_670334650a0372277aaa 89.4G Yes 78s ago 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi129 /dev/nvme2n1 ssd Linux_31751faaae429dde467e 89.4G Yes 78s ago 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi129 /dev/nvme3n1 ssd Linux_a96cbc21f96863490278 89.4G Yes 78s ago 2024-03-12T02:55:09.691 INFO:teuthology.orchestra.run.smithi079.stdout:smithi129 /dev/nvme4n1 ssd Linux_83ed5980b80677a16f4e 89.4G No 78s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-12T02:55:10.030 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:09 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.031 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:09 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.031 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:09 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.031 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:09 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.031 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:09 smithi079 ceph-mon[25794]: Deploying daemon node-exporter.a on smithi079 2024-03-12T02:55:10.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-mon[29819]: Deploying daemon node-exporter.a on smithi079 2024-03-12T02:55:10.100 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=plugin.loader t=2024-03-12T02:55:09.657039761Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-12T02:55:10.100 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=plugin.loader t=2024-03-12T02:55:09.657116633Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-12T02:55:10.100 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=secrets t=2024-03-12T02:55:09.657150892Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-12T02:55:10.100 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=query_data t=2024-03-12T02:55:09.659231552Z level=info msg="Query Service initialization" 2024-03-12T02:55:10.100 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:09 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=live.push_http t=2024-03-12T02:55:09.661692204Z level=info msg="Live Push Gateway initialization" 2024-03-12T02:55:10.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:09 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:09 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:09 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:09 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:10.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:09 smithi129 ceph-mon[30321]: Deploying daemon node-exporter.a on smithi079 2024-03-12T02:55:10.435 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-12T02:55:10.600 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=infra.usagestats.collector t=2024-03-12T02:55:10.263192953Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-12T02:55:10.600 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=provisioning.datasources t=2024-03-12T02:55:10.264184395Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-12T02:55:10.600 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=provisioning.datasources t=2024-03-12T02:55:10.274906088Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-12T02:55:10.600 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=provisioning.alerting t=2024-03-12T02:55:10.285837185Z level=info msg="starting to provision alerting" 2024-03-12T02:55:10.600 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=provisioning.alerting t=2024-03-12T02:55:10.285872239Z level=info msg="finished to provision alerting" 2024-03-12T02:55:10.600 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=ngalert.state.manager t=2024-03-12T02:55:10.286168802Z level=info msg="Warming state cache for startup" 2024-03-12T02:55:10.601 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=grafanaStorageLogger t=2024-03-12T02:55:10.286188751Z level=info msg="storage starting" 2024-03-12T02:55:10.601 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=http.server t=2024-03-12T02:55:10.288324357Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-12T02:55:10.601 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=ngalert.state.manager t=2024-03-12T02:55:10.291872231Z level=info msg="State cache has been initialized" states=0 duration=5.702943ms 2024-03-12T02:55:10.601 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=ticker t=2024-03-12T02:55:10.291924637Z level=info msg=starting first_tick=2024-03-12T02:55:20Z 2024-03-12T02:55:10.601 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=ngalert.multiorg.alertmanager t=2024-03-12T02:55:10.291933112Z level=info msg="starting MultiOrg Alertmanager" 2024-03-12T02:55:10.601 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=sqlstore.transactions t=2024-03-12T02:55:10.297178906Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-12T02:55:10.601 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=sqlstore.transactions t=2024-03-12T02:55:10.301830014Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-12T02:55:10.601 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=sqlstore.transactions t=2024-03-12T02:55:10.311270508Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-12T02:55:10.601 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=sqlstore.transactions t=2024-03-12T02:55:10.321689045Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-12T02:55:10.601 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a[42647]: logger=sqlstore.transactions t=2024-03-12T02:55:10.33183591Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-12T02:55:10.688 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:55:10.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:10 smithi079 ceph-mon[25794]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:10.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:10 smithi079 ceph-mon[25794]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:10.927 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:10 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ::ffff:172.21.15.115 - - [12/Mar/2024:02:55:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T02:55:11.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:10 smithi129 ceph-mon[30321]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:11.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:10 smithi129 ceph-mon[30321]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:11.253 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:11 smithi079 bash[54931]: Getting image source signatures 2024-03-12T02:55:11.253 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:11 smithi079 bash[54931]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-12T02:55:11.253 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:11 smithi079 bash[54931]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-12T02:55:11.253 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:11 smithi079 bash[54931]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-12T02:55:11.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-mon[29819]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:11.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:10 smithi115 ceph-mon[29819]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T02:55:12.280 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:12 smithi079 ceph-mon[25794]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:12.280 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:12 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:12.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:12 smithi115 ceph-mon[29819]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:12.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:12 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:12 smithi129 ceph-mon[30321]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:12 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:12.800 INFO:teuthology.orchestra.run.smithi079.stdout:service_type: alertmanager 2024-03-12T02:55:12.800 INFO:teuthology.orchestra.run.smithi079.stdout:service_name: alertmanager 2024-03-12T02:55:12.800 INFO:teuthology.orchestra.run.smithi079.stdout:placement: 2024-03-12T02:55:12.800 INFO:teuthology.orchestra.run.smithi079.stdout: count: 1 2024-03-12T02:55:12.800 INFO:teuthology.orchestra.run.smithi079.stdout: hosts: 2024-03-12T02:55:12.800 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi079=a 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout:status: 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout: created: '2024-03-12T02:54:05.749362Z' 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout: ports: 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout: - 9093 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout: - 9094 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout: running: 0 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout: size: 1 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout:events: 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout:- 2024-03-12T02:54:53.346504Z service:alertmanager [INFO] "service was created" 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout:--- 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout:service_type: grafana 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout:service_name: grafana 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout:placement: 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout: count: 1 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout: hosts: 2024-03-12T02:55:12.801 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi115=a 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout:spec: 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout: anonymous_access: true 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout: protocol: https 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout:status: 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout: created: '2024-03-12T02:54:08.441558Z' 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout: ports: 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout: - 3000 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout: running: 0 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout: size: 1 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout:events: 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout:- 2024-03-12T02:55:08.864193Z service:grafana [INFO] "service was created" 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout:--- 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout:service_type: mgr 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout:service_name: mgr 2024-03-12T02:55:12.802 INFO:teuthology.orchestra.run.smithi079.stdout:placement: 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: count: 3 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: hosts: 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi079=y 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi115=x 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi129=z 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:status: 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: created: '2024-03-12T02:52:32.327740Z' 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: last_refresh: '2024-03-12T02:54:32.800126Z' 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: running: 3 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: size: 3 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:--- 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:service_type: mon 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:service_name: mon 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:placement: 2024-03-12T02:55:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: count: 3 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: hosts: 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi079:172.21.15.79=a 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi115:172.21.15.115=b 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi129:172.21.15.129=c 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout:status: 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: created: '2024-03-12T02:52:04.500876Z' 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: last_refresh: '2024-03-12T02:54:32.800011Z' 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: running: 3 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: size: 3 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout:--- 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout:service_type: node-exporter 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout:service_name: node-exporter 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout:placement: 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: count: 3 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: hosts: 2024-03-12T02:55:12.804 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi079=a 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi115=b 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi129=c 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout:status: 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout: created: '2024-03-12T02:54:03.020430Z' 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout: ports: 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout: - 9100 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout: running: 0 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout: size: 3 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout:--- 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout:service_type: osd 2024-03-12T02:55:12.805 INFO:teuthology.orchestra.run.smithi079.stdout:service_name: osd 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout:unmanaged: true 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout:spec: 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout: filter_logic: AND 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout: objectstore: bluestore 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout:status: 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout: container_image_id: 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout: last_refresh: '2024-03-12T02:54:32.800207Z' 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout: running: 3 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout: size: 3 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout:--- 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout:service_type: prometheus 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout:service_name: prometheus 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout:placement: 2024-03-12T02:55:12.806 INFO:teuthology.orchestra.run.smithi079.stdout: count: 1 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout: hosts: 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi115=a 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout:status: 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout: created: '2024-03-12T02:54:00.593404Z' 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout: last_refresh: '2024-03-12T02:54:35.007009Z' 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout: ports: 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout: - 9095 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout: running: 1 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout: size: 1 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout:--- 2024-03-12T02:55:12.807 INFO:teuthology.orchestra.run.smithi079.stdout:service_type: rgw 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout:service_id: foo.a 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout:service_name: rgw.foo.a 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout:placement: 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout: count: 1 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout: hosts: 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout: - smithi079=foo.a 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout:status: 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout: created: '2024-03-12T02:54:01.622057Z' 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout: last_refresh: '2024-03-12T02:54:37.833668Z' 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout: ports: 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout: - 80 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout: running: 1 2024-03-12T02:55:12.808 INFO:teuthology.orchestra.run.smithi079.stdout: size: 1 2024-03-12T02:55:13.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:13 smithi115 ceph-mon[29819]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-12T02:55:13.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:13 smithi129 ceph-mon[30321]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-12T02:55:13.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:13 smithi079 ceph-mon[25794]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-12T02:55:13.627 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-12T02:55:13.866 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T02:55:14.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:14 smithi079 ceph-mon[25794]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:14.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:14 smithi115 ceph-mon[29819]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:14.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:14 smithi129 ceph-mon[30321]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:15.284 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:15 smithi079 bash[54931]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-12T02:55:15.559 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:15 smithi079 bash[54931]: Writing manifest to image destination 2024-03-12T02:55:16.059 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:15 smithi079 podman[54931]: 2024-03-12 02:55:15.673012956 +0000 UTC m=+5.106693490 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-12T02:55:16.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:16 smithi129 ceph-mon[30321]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:16.447 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-mon[25794]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:16.447 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 podman[54931]: 2024-03-12 02:55:16.09881613 +0000 UTC m=+5.532496668 container create 5b79667c23639fe89cbbc726e029537017811f8aa6c01d34c934ba92067d9158 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T02:55:16.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:16 smithi115 ceph-mon[29819]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T02:55:16.786 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 podman[54931]: 2024-03-12 02:55:16.562249788 +0000 UTC m=+5.995930321 container init 5b79667c23639fe89cbbc726e029537017811f8aa6c01d34c934ba92067d9158 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T02:55:16.786 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 podman[54931]: 2024-03-12 02:55:16.568167455 +0000 UTC m=+6.001847987 container start 5b79667c23639fe89cbbc726e029537017811f8aa6c01d34c934ba92067d9158 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T02:55:16.786 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.572Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.572Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=arp 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-12T02:55:16.787 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=edac 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-12T02:55:16.788 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=os 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=stat 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-12T02:55:16.789 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-12T02:55:16.790 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=time 2024-03-12T02:55:16.790 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-12T02:55:16.790 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=uname 2024-03-12T02:55:16.790 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-12T02:55:16.790 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-12T02:55:16.790 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.573Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-12T02:55:16.790 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.574Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-12T02:55:16.790 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a[55300]: ts=2024-03-12T02:55:16.574Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-12T02:55:16.790 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 bash[54931]: 5b79667c23639fe89cbbc726e029537017811f8aa6c01d34c934ba92067d9158 2024-03-12T02:55:16.790 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:55:16 smithi079 systemd[1]: Started Ceph node-exporter.a for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:55:17.118 INFO:teuthology.orchestra.run.smithi079.stdout:osd 3 44s ago - 2024-03-12T02:55:17.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:17 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:55:17.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:17 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:17.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:17 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:17.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:17 smithi129 ceph-mon[30321]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:17.378 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:17 smithi129 ceph-mon[30321]: Deploying daemon node-exporter.b on smithi115 2024-03-12T02:55:17.434 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:17 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:55:17.435 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:17 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:17.435 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:17 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:17.435 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:17 smithi079 ceph-mon[25794]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:17.435 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:17 smithi079 ceph-mon[25794]: Deploying daemon node-exporter.b on smithi115 2024-03-12T02:55:17.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:17 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T02:55:17.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:17 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:17.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:17 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:17.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:17 smithi115 ceph-mon[29819]: from='mgr.14421 172.21.15.79:0/2656231122' entity='mgr.y' 2024-03-12T02:55:17.602 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:17 smithi115 ceph-mon[29819]: Deploying daemon node-exporter.b on smithi115 2024-03-12T02:55:17.745 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-12T02:55:17.754 INFO:tasks.cephadm:Teardown begin 2024-03-12T02:55:17.754 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T02:55:17.785 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T02:55:17.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T02:55:17.855 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-12T02:55:17.855 DEBUG:teuthology.orchestra.run.smithi079:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T02:55:17.870 DEBUG:teuthology.orchestra.run.smithi115:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T02:55:17.887 DEBUG:teuthology.orchestra.run.smithi129:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T02:55:17.913 INFO:tasks.cephadm:Stopping all daemons... 2024-03-12T02:55:17.913 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-12T02:55:17.913 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.a 2024-03-12T02:55:18.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:17 smithi079 systemd[1]: Stopping Ceph mon.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:18.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a[25770]: 2024-03-12T02:55:18.196+0000 7f019cacf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T02:55:18.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a[25770]: 2024-03-12T02:55:18.196+0000 7f019cacf700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-12T02:55:18.559 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:18 smithi079 podman[55485]: 2024-03-12 02:55:18.22794457 +0000 UTC m=+0.137969881 container died 527ba2953c6bfc6e5b53f6c9031890388821d1a5675abf1bf0501325a3055026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-12T02:55:18.560 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:55:18] ENGINE Bus STOPPING 2024-03-12T02:55:18.560 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:55:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T02:55:18.560 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:55:18] ENGINE Bus STOPPED 2024-03-12T02:55:18.560 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:55:18] ENGINE Bus STARTING 2024-03-12T02:55:18.560 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:55:18] ENGINE Serving on http://:::9283 2024-03-12T02:55:18.560 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:18 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: [12/Mar/2024:02:55:18] ENGINE Bus STARTED 2024-03-12T02:55:19.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:18 smithi079 podman[55485]: 2024-03-12 02:55:18.785701762 +0000 UTC m=+0.695727066 container cleanup 527ba2953c6bfc6e5b53f6c9031890388821d1a5675abf1bf0501325a3055026 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-12T02:55:19.059 INFO:journalctl@ceph.mon.a.smithi079.stdout:Mar 12 02:55:18 smithi079 bash[55485]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-a 2024-03-12T02:55:19.100 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:19 smithi115 bash[42964]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-12T02:55:19.210 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.a.service' 2024-03-12T02:55:19.267 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:55:19.267 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-12T02:55:19.268 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-12T02:55:19.268 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.b 2024-03-12T02:55:19.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:19 smithi115 systemd[1]: Stopping Ceph mon.b for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:19.599 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:19 smithi115 bash[42964]: Getting image source signatures 2024-03-12T02:55:19.600 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:19 smithi115 bash[42964]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-12T02:55:19.600 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:19 smithi115 bash[42964]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-12T02:55:19.600 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:19 smithi115 bash[42964]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-12T02:55:21.059 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:20 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: ::ffff:172.21.15.115 - - [12/Mar/2024:02:55:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T02:55:21.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:20 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-b[29796]: 2024-03-12T02:55:20.827+0000 7f180f672700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T02:55:21.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:20 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-b[29796]: 2024-03-12T02:55:20.827+0000 7f180f672700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-12T02:55:21.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:21 smithi115 podman[43022]: 2024-03-12 02:55:21.03193444 +0000 UTC m=+1.587911114 container died 3e348d0eae3a378ffae3d0ce2825b5f8c28fc80a48bf54f9e4004f1b4fc2a86a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.build-date=20240307) 2024-03-12T02:55:22.349 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:22 smithi115 bash[42964]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-12T02:55:22.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:22 smithi115 podman[43022]: 2024-03-12 02:55:22.042480449 +0000 UTC m=+2.598457111 container cleanup 3e348d0eae3a378ffae3d0ce2825b5f8c28fc80a48bf54f9e4004f1b4fc2a86a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307) 2024-03-12T02:55:22.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:22 smithi115 bash[43022]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-b 2024-03-12T02:55:22.802 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:22 smithi115 bash[42964]: Writing manifest to image destination 2024-03-12T02:55:22.803 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 12 02:55:22 smithi115 podman[43043]: 2024-03-12 02:55:22.395251724 +0000 UTC m=+1.362584238 container remove 3e348d0eae3a378ffae3d0ce2825b5f8c28fc80a48bf54f9e4004f1b4fc2a86a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T02:55:23.094 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:22 smithi115 podman[42964]: 2024-03-12 02:55:22.750923316 +0000 UTC m=+3.779683567 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-12T02:55:23.094 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:22 smithi115 podman[42964]: 2024-03-12 02:55:22.857486169 +0000 UTC m=+3.886246420 container create 23211177ca3d112defcd7eff3db47be551f1c9874af7379aa67a468f5e1740c5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T02:55:23.094 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 podman[42964]: 2024-03-12 02:55:23.092453841 +0000 UTC m=+4.121214091 container init 23211177ca3d112defcd7eff3db47be551f1c9874af7379aa67a468f5e1740c5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T02:55:23.220 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.b.service' 2024-03-12T02:55:23.270 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:55:23.270 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-12T02:55:23.271 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-12T02:55:23.271 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.c 2024-03-12T02:55:23.350 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 podman[42964]: 2024-03-12 02:55:23.098083512 +0000 UTC m=+4.126843771 container start 23211177ca3d112defcd7eff3db47be551f1c9874af7379aa67a468f5e1740c5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T02:55:23.350 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.101Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-12T02:55:23.350 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.101Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-12T02:55:23.350 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.101Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-12T02:55:23.350 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.101Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T02:55:23.350 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-12T02:55:23.350 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-12T02:55:23.350 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-12T02:55:23.350 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=arp 2024-03-12T02:55:23.351 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-12T02:55:23.351 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-12T02:55:23.351 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-12T02:55:23.351 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-12T02:55:23.351 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-12T02:55:23.351 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-12T02:55:23.351 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-12T02:55:23.351 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=edac 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-12T02:55:23.352 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=os 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-12T02:55:23.353 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=stat 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=time 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=uname 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.102Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.103Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b[43125]: ts=2024-03-12T02:55:23.103Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 bash[42964]: 23211177ca3d112defcd7eff3db47be551f1c9874af7379aa67a468f5e1740c5 2024-03-12T02:55:23.354 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:55:23 smithi115 systemd[1]: Started Ceph node-exporter.b for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:55:23.565 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:23 smithi129 systemd[1]: Stopping Ceph mon.c for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:23 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-c[30283]: 2024-03-12T02:55:23.561+0000 7ff700a19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T02:55:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:23 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-c[30283]: 2024-03-12T02:55:23.561+0000 7ff700a19700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-12T02:55:23.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:23 smithi129 podman[44022]: 2024-03-12 02:55:23.695923582 +0000 UTC m=+0.230862508 container died 4c148043ea863a6705118901b04ae2725a309f4fa5ceb692e19e135f34b5b6e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-c, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-12T02:55:24.337 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:24 smithi129 podman[44022]: 2024-03-12 02:55:24.109212892 +0000 UTC m=+0.644151860 container cleanup 4c148043ea863a6705118901b04ae2725a309f4fa5ceb692e19e135f34b5b6e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-2b0101e, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-12T02:55:24.337 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:24 smithi129 bash[44022]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-c 2024-03-12T02:55:24.581 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.c.service' 2024-03-12T02:55:24.604 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:24 smithi129 podman[44041]: 2024-03-12 02:55:24.335505052 +0000 UTC m=+0.640920166 container remove 4c148043ea863a6705118901b04ae2725a309f4fa5ceb692e19e135f34b5b6e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mon-c, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-12T02:55:24.604 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:24 smithi129 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-12T02:55:24.604 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:24 smithi129 systemd[1]: Stopped Ceph mon.c for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:55:24.604 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 12 02:55:24 smithi129 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mon.c.service: Consumed 3.546s CPU time. 2024-03-12T02:55:24.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:55:24.625 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-12T02:55:24.625 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-12T02:55:24.625 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.y 2024-03-12T02:55:24.910 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:24 smithi079 systemd[1]: Stopping Ceph mgr.y for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:25.258 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:24 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:55:24.907+0000 7fb08bd1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T02:55:25.258 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:24 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y[26051]: 2024-03-12T02:55:24.907+0000 7fb08bd1e700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-12T02:55:25.258 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:24 smithi079 podman[55627]: 2024-03-12 02:55:24.94031057 +0000 UTC m=+0.118391485 container died 13cc77d2a103ce63930492462bead608dab34f9f1da5e41d54822cf30750de7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-2b0101e) 2024-03-12T02:55:25.559 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:25 smithi079 podman[55627]: 2024-03-12 02:55:25.430841453 +0000 UTC m=+0.608922367 container cleanup 13cc77d2a103ce63930492462bead608dab34f9f1da5e41d54822cf30750de7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T02:55:25.559 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Mar 12 02:55:25 smithi079 bash[55627]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-y 2024-03-12T02:55:25.876 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-12T02:55:25.915 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:55:25.915 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-12T02:55:25.916 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-12T02:55:25.916 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.x 2024-03-12T02:55:26.247 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:55:25 smithi115 systemd[1]: Stopping Ceph mgr.x for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:26.532 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:55:26 smithi115 podman[43224]: 2024-03-12 02:55:26.245289283 +0000 UTC m=+0.134910641 container died 43d1f31507b4de219bcc9a4b88ededd8f5d8df86ba29f4c662bbbc0f271080d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-12T02:55:26.849 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:55:26 smithi115 podman[43224]: 2024-03-12 02:55:26.612782256 +0000 UTC m=+0.502403602 container cleanup 43d1f31507b4de219bcc9a4b88ededd8f5d8df86ba29f4c662bbbc0f271080d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-12T02:55:26.849 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 12 02:55:26 smithi115 bash[43224]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-x 2024-03-12T02:55:27.041 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-12T02:55:27.079 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:55:27.079 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-12T02:55:27.079 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-12T02:55:27.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.z 2024-03-12T02:55:27.378 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:55:27 smithi129 systemd[1]: Stopping Ceph mgr.z for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:27.641 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:55:27 smithi129 podman[44155]: 2024-03-12 02:55:27.378996499 +0000 UTC m=+0.108155134 container died bbfacb7d3ec9929aaa0ebc08938981fd8c98cf157f1fb00ccf71d7c5760cd552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T02:55:27.916 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:55:27 smithi129 podman[44155]: 2024-03-12 02:55:27.696344258 +0000 UTC m=+0.425502901 container cleanup bbfacb7d3ec9929aaa0ebc08938981fd8c98cf157f1fb00ccf71d7c5760cd552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-12T02:55:27.916 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 12 02:55:27 smithi129 bash[44155]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-mgr-z 2024-03-12T02:55:28.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-12T02:55:28.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:55:28.093 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-12T02:55:28.093 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-12T02:55:28.094 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.0 2024-03-12T02:55:28.423 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:28 smithi079 systemd[1]: Stopping Ceph osd.0 for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:28.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:28 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0[39977]: 2024-03-12T02:55:28.419+0000 7f6d5b483700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T02:55:28.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:28 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0[39977]: 2024-03-12T02:55:28.419+0000 7f6d5b483700 -1 osd.0 32 *** Got signal Terminated *** 2024-03-12T02:55:28.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:28 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0[39977]: 2024-03-12T02:55:28.419+0000 7f6d5b483700 -1 osd.0 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T02:55:33.996 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:33 smithi079 podman[55768]: 2024-03-12 02:55:33.729195733 +0000 UTC m=+5.445360996 container died cc6516580864bcaf24405d3559feb6cc9fb4cd5651c36d5b8adf1dd598f9f0d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-12T02:55:34.309 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:34 smithi079 podman[55768]: 2024-03-12 02:55:34.1588247 +0000 UTC m=+5.874989967 container cleanup cc6516580864bcaf24405d3559feb6cc9fb4cd5651c36d5b8adf1dd598f9f0d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-12T02:55:34.309 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:34 smithi079 bash[55768]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0 2024-03-12T02:55:34.768 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:34 smithi079 podman[55790]: 2024-03-12 02:55:34.334957496 +0000 UTC m=+0.606006687 container remove cc6516580864bcaf24405d3559feb6cc9fb4cd5651c36d5b8adf1dd598f9f0d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_CLEAN=True) 2024-03-12T02:55:35.059 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:34 smithi079 podman[55889]: 2024-03-12 02:55:34.669153288 +0000 UTC m=+0.017718614 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T02:55:35.060 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:34 smithi079 podman[55889]: 2024-03-12 02:55:34.777226094 +0000 UTC m=+0.125791412 container create ea3f2899c9972cbb88cd488c60a85ed7cc5e3490a997d4c03fa43ae6c5ee50a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T02:55:35.060 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:34 smithi079 podman[55889]: 2024-03-12 02:55:34.996450622 +0000 UTC m=+0.345015948 container init ea3f2899c9972cbb88cd488c60a85ed7cc5e3490a997d4c03fa43ae6c5ee50a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T02:55:35.060 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:35 smithi079 podman[55889]: 2024-03-12 02:55:35.002286995 +0000 UTC m=+0.350852319 container start ea3f2899c9972cbb88cd488c60a85ed7cc5e3490a997d4c03fa43ae6c5ee50a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T02:55:35.342 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:35 smithi079 podman[55889]: 2024-03-12 02:55:35.095822711 +0000 UTC m=+0.444388038 container attach ea3f2899c9972cbb88cd488c60a85ed7cc5e3490a997d4c03fa43ae6c5ee50a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-03-12T02:55:35.342 INFO:journalctl@ceph.osd.0.smithi079.stdout:Mar 12 02:55:35 smithi079 podman[55889]: 2024-03-12 02:55:35.300775176 +0000 UTC m=+0.649340505 container died ea3f2899c9972cbb88cd488c60a85ed7cc5e3490a997d4c03fa43ae6c5ee50a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-2b0101e, ceph=True, GIT_CLEAN=True) 2024-03-12T02:55:36.067 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.0.service' 2024-03-12T02:55:36.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:55:36.108 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-12T02:55:36.108 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-12T02:55:36.108 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.1 2024-03-12T02:55:36.411 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:36 smithi115 systemd[1]: Stopping Ceph osd.1 for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:36.849 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:36 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1[34422]: 2024-03-12T02:55:36.408+0000 7f19e6890700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T02:55:36.850 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:36 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1[34422]: 2024-03-12T02:55:36.409+0000 7f19e6890700 -1 osd.1 32 *** Got signal Terminated *** 2024-03-12T02:55:36.850 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:36 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1[34422]: 2024-03-12T02:55:36.409+0000 7f19e6890700 -1 osd.1 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T02:55:42.192 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:41 smithi115 podman[43355]: 2024-03-12 02:55:41.89344079 +0000 UTC m=+5.590825046 container died 9128a0afb07d35750d5f2d433b28ec975f48e97a9a28b2d9aa5193c599d995c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T02:55:42.517 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:42 smithi115 podman[43355]: 2024-03-12 02:55:42.314248092 +0000 UTC m=+6.011632349 container cleanup 9128a0afb07d35750d5f2d433b28ec975f48e97a9a28b2d9aa5193c599d995c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-12T02:55:42.517 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:42 smithi115 bash[43355]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1 2024-03-12T02:55:42.849 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:42 smithi115 podman[43377]: 2024-03-12 02:55:42.515336888 +0000 UTC m=+0.616566635 container remove 9128a0afb07d35750d5f2d433b28ec975f48e97a9a28b2d9aa5193c599d995c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T02:55:43.245 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:42 smithi115 podman[43473]: 2024-03-12 02:55:42.835699949 +0000 UTC m=+0.018352122 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T02:55:43.245 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:42 smithi115 podman[43473]: 2024-03-12 02:55:42.941090242 +0000 UTC m=+0.123742475 container create 6673c4d0d2e8c5347050e3e18f9360dd88f9274018a80600156f9a71e796d18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-12T02:55:43.245 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:43 smithi115 podman[43473]: 2024-03-12 02:55:43.159857439 +0000 UTC m=+0.342509599 container init 6673c4d0d2e8c5347050e3e18f9360dd88f9274018a80600156f9a71e796d18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-12T02:55:43.245 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:43 smithi115 podman[43473]: 2024-03-12 02:55:43.165088999 +0000 UTC m=+0.347741162 container start 6673c4d0d2e8c5347050e3e18f9360dd88f9274018a80600156f9a71e796d18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-12T02:55:43.245 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:43 smithi115 podman[43473]: 2024-03-12 02:55:43.242987654 +0000 UTC m=+0.425639827 container attach 6673c4d0d2e8c5347050e3e18f9360dd88f9274018a80600156f9a71e796d18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T02:55:43.599 INFO:journalctl@ceph.osd.1.smithi115.stdout:Mar 12 02:55:43 smithi115 podman[43473]: 2024-03-12 02:55:43.457725663 +0000 UTC m=+0.640377824 container died 6673c4d0d2e8c5347050e3e18f9360dd88f9274018a80600156f9a71e796d18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-12T02:55:44.032 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.1.service' 2024-03-12T02:55:44.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:55:44.073 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-12T02:55:44.073 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-12T02:55:44.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.2 2024-03-12T02:55:44.354 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:44 smithi129 systemd[1]: Stopping Ceph osd.2 for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:44.628 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:44 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2[35611]: 2024-03-12T02:55:44.351+0000 7f71ae08d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T02:55:44.628 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:44 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2[35611]: 2024-03-12T02:55:44.351+0000 7f71ae08d700 -1 osd.2 32 *** Got signal Terminated *** 2024-03-12T02:55:44.628 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:44 smithi129 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2[35611]: 2024-03-12T02:55:44.351+0000 7f71ae08d700 -1 osd.2 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T02:55:49.968 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:49 smithi129 podman[44303]: 2024-03-12 02:55:49.70528705 +0000 UTC m=+5.438930548 container died 795804c2be19abae8c1ec7d861863575c6e49edcff335cdd270c2785367e7ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-12T02:55:50.378 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:50 smithi129 podman[44303]: 2024-03-12 02:55:50.040504928 +0000 UTC m=+5.774148407 container cleanup 795804c2be19abae8c1ec7d861863575c6e49edcff335cdd270c2785367e7ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-12T02:55:50.378 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:50 smithi129 bash[44303]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2 2024-03-12T02:55:50.378 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:50 smithi129 podman[44325]: 2024-03-12 02:55:50.208085104 +0000 UTC m=+0.501536129 container remove 795804c2be19abae8c1ec7d861863575c6e49edcff335cdd270c2785367e7ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-12T02:55:50.878 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:50 smithi129 podman[44419]: 2024-03-12 02:55:50.501642934 +0000 UTC m=+0.018025040 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T02:55:50.878 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:50 smithi129 podman[44419]: 2024-03-12 02:55:50.600678066 +0000 UTC m=+0.117060214 container create 5f2824c4982a25d57c5a5d552c00c2f94f1c0dd244c53a82700d6624afc9da08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-2b0101e) 2024-03-12T02:55:50.878 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:50 smithi129 podman[44419]: 2024-03-12 02:55:50.828942656 +0000 UTC m=+0.345324769 container init 5f2824c4982a25d57c5a5d552c00c2f94f1c0dd244c53a82700d6624afc9da08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e) 2024-03-12T02:55:50.878 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:50 smithi129 podman[44419]: 2024-03-12 02:55:50.834410031 +0000 UTC m=+0.350792141 container start 5f2824c4982a25d57c5a5d552c00c2f94f1c0dd244c53a82700d6624afc9da08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T02:55:51.378 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:50 smithi129 podman[44419]: 2024-03-12 02:55:50.911198747 +0000 UTC m=+0.427580864 container attach 5f2824c4982a25d57c5a5d552c00c2f94f1c0dd244c53a82700d6624afc9da08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-12T02:55:51.378 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:51 smithi129 podman[44419]: 2024-03-12 02:55:51.123937049 +0000 UTC m=+0.640319159 container died 5f2824c4982a25d57c5a5d552c00c2f94f1c0dd244c53a82700d6624afc9da08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True) 2024-03-12T02:55:51.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.2.service' 2024-03-12T02:55:51.676 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:51 smithi129 podman[44465]: 2024-03-12 02:55:51.622547387 +0000 UTC m=+0.493660370 container remove 5f2824c4982a25d57c5a5d552c00c2f94f1c0dd244c53a82700d6624afc9da08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-osd-2-deactivate, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T02:55:51.676 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:51 smithi129 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-12T02:55:51.676 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:51 smithi129 systemd[1]: Stopped Ceph osd.2 for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:55:51.676 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 12 02:55:51 smithi129 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@osd.2.service: Consumed 3.821s CPU time. 2024-03-12T02:55:51.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:55:51.690 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-12T02:55:51.690 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-12T02:55:51.690 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-12T02:55:52.059 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:55:51 smithi079 systemd[1]: Stopping Ceph rgw.foo.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:55:52.559 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:55:52 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-rgw-foo-a[46008]: 2024-03-12T02:55:52.069+0000 7f8234d11700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T02:55:52.559 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:55:52 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-rgw-foo-a[46008]: 2024-03-12T02:55:52.069+0000 7f8243589ac0 -1 shutting down 2024-03-12T02:56:02.417 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:56:02 smithi079 bash[56026]: time="2024-03-12T02:56:02Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-12T02:56:02.417 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:56:02 smithi079 podman[56026]: 2024-03-12 02:56:02.089903144 +0000 UTC m=+10.202867186 container died 7d29761e6c9d3fe6a7d3a987af555fe2292d3873bfd6d7067f609bb00f2b485a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True) 2024-03-12T02:56:02.809 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:56:02 smithi079 podman[56026]: 2024-03-12 02:56:02.581070271 +0000 UTC m=+10.694034355 container cleanup 7d29761e6c9d3fe6a7d3a987af555fe2292d3873bfd6d7067f609bb00f2b485a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T02:56:02.809 INFO:journalctl@ceph.rgw.foo.a.smithi079.stdout:Mar 12 02:56:02 smithi079 bash[56026]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-12T02:56:02.951 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-12T02:56:02.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:56:02.988 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-12T02:56:02.988 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-12T02:56:02.988 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@prometheus.a 2024-03-12T02:56:03.349 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 systemd[1]: Stopping Ceph prometheus.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:56:03.757 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T02:56:03.757 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-12T02:56:03.757 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-12T02:56:03.757 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T02:56:03.757 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-12T02:56:03.757 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T02:56:03.757 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-12T02:56:03.757 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-12T02:56:03.757 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-12T02:56:03.758 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T02:56:03.758 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-12T02:56:03.758 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a[40772]: ts=2024-03-12T02:56:03.465Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-12T02:56:03.758 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 podman[43615]: 2024-03-12 02:56:03.47569928 +0000 UTC m=+0.293905416 container died 85e505665ae840b46cdcf4f7476ef67c2a5944e4ac5f1f5a842caaa2be7cac43 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T02:56:04.089 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 podman[43615]: 2024-03-12 02:56:03.845543588 +0000 UTC m=+0.663749731 container cleanup 85e505665ae840b46cdcf4f7476ef67c2a5944e4ac5f1f5a842caaa2be7cac43 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T02:56:04.089 INFO:journalctl@ceph.prometheus.a.smithi115.stdout:Mar 12 02:56:03 smithi115 bash[43615]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T02:56:04.263 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-12T02:56:04.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T02:56:04.299 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-12T02:56:04.299 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T02:56:04.485 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:56:05.809 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:56:05 smithi079 systemd[1]: Stopping Ceph alertmanager.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:56:06.122 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:56:05 smithi079 ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a[53345]: ts=2024-03-12T02:56:05.838Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-12T02:56:06.122 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:56:05 smithi079 podman[56382]: 2024-03-12 02:56:05.869208342 +0000 UTC m=+0.141018231 container died 03a2b1dc0bb16ba2a28be486f1c371da8e6007c9e067f6d675923f91fbc906c6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T02:56:06.479 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:56:06 smithi079 podman[56382]: 2024-03-12 02:56:06.285433116 +0000 UTC m=+0.557243013 container cleanup 03a2b1dc0bb16ba2a28be486f1c371da8e6007c9e067f6d675923f91fbc906c6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T02:56:06.479 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:56:06 smithi079 bash[56382]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a 2024-03-12T02:56:06.809 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:56:06 smithi079 podman[56401]: 2024-03-12 02:56:06.478115081 +0000 UTC m=+0.613731372 container remove 03a2b1dc0bb16ba2a28be486f1c371da8e6007c9e067f6d675923f91fbc906c6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T02:56:06.809 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:56:06 smithi079 podman[56401]: 2024-03-12 02:56:06.553735259 +0000 UTC m=+0.689351557 volume remove 498ce7e34d941e2b48fecbda161543519d83d51e0b3e1a39d031cda646bac167 2024-03-12T02:56:07.305 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:56:07 smithi079 systemd[1]: Stopping Ceph node-exporter.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:56:07.305 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:56:06 smithi079 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-12T02:56:07.305 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Mar 12 02:56:06 smithi079 systemd[1]: Stopped Ceph alertmanager.a for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:56:07.559 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:56:07 smithi079 podman[56519]: 2024-03-12 02:56:07.303129252 +0000 UTC m=+0.118342105 container died 5b79667c23639fe89cbbc726e029537017811f8aa6c01d34c934ba92067d9158 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T02:56:08.059 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:56:07 smithi079 podman[56519]: 2024-03-12 02:56:07.714313865 +0000 UTC m=+0.529526717 container cleanup 5b79667c23639fe89cbbc726e029537017811f8aa6c01d34c934ba92067d9158 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T02:56:08.059 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:56:07 smithi079 bash[56519]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a 2024-03-12T02:56:08.059 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:56:07 smithi079 podman[56537]: 2024-03-12 02:56:07.881805204 +0000 UTC m=+0.581758444 container remove 5b79667c23639fe89cbbc726e029537017811f8aa6c01d34c934ba92067d9158 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T02:56:08.059 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:56:07 smithi079 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T02:56:08.521 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:56:08 smithi079 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-12T02:56:08.521 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:56:08 smithi079 systemd[1]: Stopped Ceph node-exporter.a for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:56:08.521 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Mar 12 02:56:08 smithi079 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 1.846s CPU time. 2024-03-12T02:56:08.711 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T02:56:08.903 INFO:teuthology.orchestra.run.smithi115.stdout:Deleting cluster with fsid: 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:56:09.849 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:56:09 smithi115 systemd[1]: Stopping Ceph grafana.a for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:56:10.267 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:56:10 smithi115 podman[43951]: 2024-03-12 02:56:10.003945561 +0000 UTC m=+0.114481113 container died 776fa070fbb21c274e66ab9bce5202cbda5f6d5ad32526a0e7b1cafe18c5e319 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, vendor=Red Hat, Inc., release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, name=ubi8, vcs-type=git, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-12T02:56:10.599 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:56:10 smithi115 podman[43951]: 2024-03-12 02:56:10.348299319 +0000 UTC m=+0.458834871 container cleanup 776fa070fbb21c274e66ab9bce5202cbda5f6d5ad32526a0e7b1cafe18c5e319 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, maintainer=Paul Cuzner , release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, version=8.5, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-12T02:56:10.600 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:56:10 smithi115 bash[43951]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a 2024-03-12T02:56:10.968 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:56:10 smithi115 podman[43970]: 2024-03-12 02:56:10.632427511 +0000 UTC m=+0.629347907 container remove 776fa070fbb21c274e66ab9bce5202cbda5f6d5ad32526a0e7b1cafe18c5e319 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-grafana-a, description=Ceph Grafana Container, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vendor=Red Hat, Inc., io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.openshift.tags=base rhel8, maintainer=Paul Cuzner ) 2024-03-12T02:56:10.968 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:56:10 smithi115 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T02:56:10.968 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:56:10 smithi115 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-12T02:56:10.968 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:56:10 smithi115 systemd[1]: Stopped Ceph grafana.a for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:56:10.968 INFO:journalctl@ceph.grafana.a.smithi115.stdout:Mar 12 02:56:10 smithi115 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@grafana.a.service: Consumed 2.690s CPU time. 2024-03-12T02:56:11.349 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:56:10 smithi115 systemd[1]: Stopping Ceph node-exporter.b for 0b28b95e-e01b-11ee-95c8-87774f69a715... 2024-03-12T02:56:11.349 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:56:11 smithi115 podman[44085]: 2024-03-12 02:56:11.206773916 +0000 UTC m=+0.106250231 container died 23211177ca3d112defcd7eff3db47be551f1c9874af7379aa67a468f5e1740c5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T02:56:11.829 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:56:11 smithi115 podman[44085]: 2024-03-12 02:56:11.551756052 +0000 UTC m=+0.451232415 container cleanup 23211177ca3d112defcd7eff3db47be551f1c9874af7379aa67a468f5e1740c5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T02:56:11.829 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:56:11 smithi115 bash[44085]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b 2024-03-12T02:56:12.099 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:56:11 smithi115 podman[44104]: 2024-03-12 02:56:11.827469157 +0000 UTC m=+0.625829293 container remove 23211177ca3d112defcd7eff3db47be551f1c9874af7379aa67a468f5e1740c5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-0b28b95e-e01b-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T02:56:12.099 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:56:11 smithi115 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-12T02:56:12.100 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:56:11 smithi115 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-12T02:56:12.100 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:56:11 smithi115 systemd[1]: Stopped Ceph node-exporter.b for 0b28b95e-e01b-11ee-95c8-87774f69a715. 2024-03-12T02:56:12.100 INFO:journalctl@ceph.node-exporter.b.smithi115.stdout:Mar 12 02:56:11 smithi115 systemd[1]: ceph-0b28b95e-e01b-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 1.721s CPU time. 2024-03-12T02:56:12.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T02:56:12.825 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:56:13.997 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T02:56:14.032 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T02:56:14.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T02:56:14.092 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-12T02:56:14.093 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/remote/smithi079/crash 2024-03-12T02:56:14.094 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/crash -- . 2024-03-12T02:56:14.146 INFO:teuthology.orchestra.run.smithi079.stderr:tar: /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T02:56:14.146 INFO:teuthology.orchestra.run.smithi079.stderr:tar: Error is not recoverable: exiting now 2024-03-12T02:56:14.148 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/remote/smithi115/crash 2024-03-12T02:56:14.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/crash -- . 2024-03-12T02:56:14.487 INFO:teuthology.orchestra.run.smithi115.stderr:tar: /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T02:56:14.488 INFO:teuthology.orchestra.run.smithi115.stderr:tar: Error is not recoverable: exiting now 2024-03-12T02:56:14.490 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/remote/smithi129/crash 2024-03-12T02:56:14.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/crash -- . 2024-03-12T02:56:14.530 INFO:teuthology.orchestra.run.smithi129.stderr:tar: /var/lib/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T02:56:14.530 INFO:teuthology.orchestra.run.smithi129.stderr:tar: Error is not recoverable: exiting now 2024-03-12T02:56:14.531 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-12T02:56:14.531 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-12T02:56:14.564 INFO:teuthology.orchestra.run.smithi079.stdout:2024-03-12T02:52:23.500567+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T02:56:14.565 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-12T02:56:14.565 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[SEC\]' /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-12T02:56:14.637 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]' /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-12T02:56:14.708 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[WRN\]' /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-12T02:56:14.777 INFO:teuthology.orchestra.run.smithi079.stdout:2024-03-12T02:52:23.500567+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T02:56:14.778 INFO:tasks.cephadm:Compressing logs... 2024-03-12T02:56:14.778 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T02:56:14.820 DEBUG:teuthology.orchestra.run.smithi115:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T02:56:14.823 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T02:56:14.845 INFO:teuthology.orchestra.run.smithi079.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T02:56:14.845 INFO:teuthology.orchestra.run.smithi079.stderr:: No such file or directory 2024-03-12T02:56:14.845 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-12T02:56:14.845 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T02:56:14.845 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mon.a.log: -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-12T02:56:14.848 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-12T02:56:14.848 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-12T02:56:14.848 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mgr.y.log: 57.7% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-12T02:56:14.848 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T02:56:14.848 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log 2024-03-12T02:56:14.848 INFO:teuthology.orchestra.run.smithi079.stderr: 89.4% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T02:56:14.848 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T02:56:14.848 INFO:teuthology.orchestra.run.smithi079.stderr: 90.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T02:56:14.848 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log: 82.7% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T02:56:14.848 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-volume.log: 84.6% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr:: No such file or directory 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.audit.log: /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr:gzip/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mgr.x.log: -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr: 89.4% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr: 81.4% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-12T02:56:14.851 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-volume.log: 84.0% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T02:56:14.852 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mon.b.log: 81.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T02:56:14.852 INFO:teuthology.orchestra.run.smithi079.stderr: 90.5% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T02:56:14.853 INFO:teuthology.orchestra.run.smithi129.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T02:56:14.853 INFO:teuthology.orchestra.run.smithi129.stderr:: No such file or directory 2024-03-12T02:56:14.853 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T02:56:14.853 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-12T02:56:14.853 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T02:56:14.853 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.audit.log: /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log 2024-03-12T02:56:14.854 INFO:teuthology.orchestra.run.smithi129.stderr: 90.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T02:56:14.854 INFO:teuthology.orchestra.run.smithi115.stderr: 90.5% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T02:56:14.854 INFO:teuthology.orchestra.run.smithi115.stderr: 90.2% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-12T02:56:14.855 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-12T02:56:14.856 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.cephadm.log: /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T02:56:14.856 INFO:teuthology.orchestra.run.smithi129.stderr: 81.4% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T02:56:14.856 INFO:teuthology.orchestra.run.smithi129.stderr: 89.6% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T02:56:14.856 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-12T02:56:14.856 INFO:teuthology.orchestra.run.smithi129.stderr: 84.2% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T02:56:14.856 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.2% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-12T02:56:14.857 INFO:teuthology.orchestra.run.smithi129.stderr: 90.5% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T02:56:14.899 INFO:teuthology.orchestra.run.smithi115.stderr: 92.2% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-12T02:56:14.907 INFO:teuthology.orchestra.run.smithi129.stderr: 92.2% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-12T02:56:14.914 INFO:teuthology.orchestra.run.smithi079.stderr: 89.0% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-12T02:56:15.015 INFO:teuthology.orchestra.run.smithi079.stderr: 91.4% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-12T02:56:15.126 INFO:teuthology.orchestra.run.smithi079.stderr: 94.2% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-12T02:56:15.128 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-12T02:56:15.128 INFO:teuthology.orchestra.run.smithi079.stderr:real 0m0.295s 2024-03-12T02:56:15.128 INFO:teuthology.orchestra.run.smithi079.stderr:user 0m0.523s 2024-03-12T02:56:15.128 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m0.030s 2024-03-12T02:56:15.141 INFO:teuthology.orchestra.run.smithi129.stderr: 94.2% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-12T02:56:15.143 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-12T02:56:15.143 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m0.305s 2024-03-12T02:56:15.143 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m0.352s 2024-03-12T02:56:15.143 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m0.035s 2024-03-12T02:56:15.164 INFO:teuthology.orchestra.run.smithi115.stderr: 94.2% -- replaced with /var/log/ceph/0b28b95e-e01b-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-12T02:56:15.166 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-12T02:56:15.166 INFO:teuthology.orchestra.run.smithi115.stderr:real 0m0.332s 2024-03-12T02:56:15.166 INFO:teuthology.orchestra.run.smithi115.stderr:user 0m0.382s 2024-03-12T02:56:15.166 INFO:teuthology.orchestra.run.smithi115.stderr:sys 0m0.028s 2024-03-12T02:56:15.167 INFO:tasks.cephadm:Archiving logs... 2024-03-12T02:56:15.167 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/remote/smithi079/log 2024-03-12T02:56:15.168 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T02:56:15.285 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/remote/smithi115/log 2024-03-12T02:56:15.286 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T02:56:15.373 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/remote/smithi129/log 2024-03-12T02:56:15.374 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T02:56:15.461 INFO:tasks.cephadm:Removing cluster... 2024-03-12T02:56:15.461 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 --force 2024-03-12T02:56:15.642 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:56:15.757 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 --force 2024-03-12T02:56:15.938 INFO:teuthology.orchestra.run.smithi115.stdout:Deleting cluster with fsid: 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:56:16.051 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0b28b95e-e01b-11ee-95c8-87774f69a715 --force 2024-03-12T02:56:16.241 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: 0b28b95e-e01b-11ee-95c8-87774f69a715 2024-03-12T02:56:16.357 INFO:tasks.cephadm:Removing cephadm ... 2024-03-12T02:56:16.357 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T02:56:16.373 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T02:56:16.388 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T02:56:16.403 INFO:tasks.cephadm:Teardown complete 2024-03-12T02:56:16.403 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-12T02:56:16.413 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi079:/dev/vg_nvme/lv_1... 2024-03-12T02:56:16.413 DEBUG:teuthology.orchestra.run.smithi079:> sudo nvme disconnect -n lv_1 2024-03-12T02:56:16.561 INFO:teuthology.orchestra.run.smithi079.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-12T02:56:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:16.563 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi079:/dev/vg_nvme/lv_2... 2024-03-12T02:56:16.563 DEBUG:teuthology.orchestra.run.smithi079:> sudo nvme disconnect -n lv_2 2024-03-12T02:56:16.712 INFO:teuthology.orchestra.run.smithi079.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-12T02:56:16.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:16.714 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi079:/dev/vg_nvme/lv_3... 2024-03-12T02:56:16.714 DEBUG:teuthology.orchestra.run.smithi079:> sudo nvme disconnect -n lv_3 2024-03-12T02:56:16.912 INFO:teuthology.orchestra.run.smithi079.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-12T02:56:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:16.914 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi079:/dev/vg_nvme/lv_4... 2024-03-12T02:56:16.914 DEBUG:teuthology.orchestra.run.smithi079:> sudo nvme disconnect -n lv_4 2024-03-12T02:56:17.087 INFO:teuthology.orchestra.run.smithi079.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-12T02:56:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:17.089 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-12T02:56:17.089 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/scratch_devs 2024-03-12T02:56:17.119 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi115:/dev/vg_nvme/lv_1... 2024-03-12T02:56:17.119 DEBUG:teuthology.orchestra.run.smithi115:> sudo nvme disconnect -n lv_1 2024-03-12T02:56:17.272 INFO:teuthology.orchestra.run.smithi115.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-12T02:56:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:17.274 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi115:/dev/vg_nvme/lv_2... 2024-03-12T02:56:17.274 DEBUG:teuthology.orchestra.run.smithi115:> sudo nvme disconnect -n lv_2 2024-03-12T02:56:17.414 INFO:teuthology.orchestra.run.smithi115.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-12T02:56:17.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:17.416 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi115:/dev/vg_nvme/lv_3... 2024-03-12T02:56:17.416 DEBUG:teuthology.orchestra.run.smithi115:> sudo nvme disconnect -n lv_3 2024-03-12T02:56:17.556 INFO:teuthology.orchestra.run.smithi115.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-12T02:56:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:17.558 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi115:/dev/vg_nvme/lv_4... 2024-03-12T02:56:17.558 DEBUG:teuthology.orchestra.run.smithi115:> sudo nvme disconnect -n lv_4 2024-03-12T02:56:17.708 INFO:teuthology.orchestra.run.smithi115.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-12T02:56:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:17.710 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-12T02:56:17.710 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/scratch_devs 2024-03-12T02:56:17.740 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi129:/dev/vg_nvme/lv_1... 2024-03-12T02:56:17.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo nvme disconnect -n lv_1 2024-03-12T02:56:17.889 INFO:teuthology.orchestra.run.smithi129.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-12T02:56:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:17.891 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi129:/dev/vg_nvme/lv_2... 2024-03-12T02:56:17.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo nvme disconnect -n lv_2 2024-03-12T02:56:18.038 INFO:teuthology.orchestra.run.smithi129.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-12T02:56:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:18.040 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi129:/dev/vg_nvme/lv_3... 2024-03-12T02:56:18.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo nvme disconnect -n lv_3 2024-03-12T02:56:18.187 INFO:teuthology.orchestra.run.smithi129.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-12T02:56:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:18.189 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi129:/dev/vg_nvme/lv_4... 2024-03-12T02:56:18.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo nvme disconnect -n lv_4 2024-03-12T02:56:18.341 INFO:teuthology.orchestra.run.smithi129.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-12T02:56:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:18.343 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-12T02:56:18.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/scratch_devs 2024-03-12T02:56:18.373 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-12T02:56:18.382 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-12T02:56:18.382 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T02:56:18.385 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T02:56:18.388 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T02:56:18.398 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-03-12T02:56:18.402 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-03-12T02:56:18.402 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T02:56:18.403 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-03-12T02:56:18.403 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 -165us[ -165us] +/- 95ms 2024-03-12T02:56:18.403 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -1427us[-1429us] +/- 65ms 2024-03-12T02:56:18.403 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +1609us[+1608us] +/- 63ms 2024-03-12T02:56:18.403 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:56:18.406 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T02:56:18.407 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-03-12T02:56:18.407 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 -175us[ -175us] +/- 95ms 2024-03-12T02:56:18.407 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -1446us[-1453us] +/- 65ms 2024-03-12T02:56:18.407 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +1604us[+1604us] +/- 63ms 2024-03-12T02:56:18.407 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:56:18.429 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-12T02:56:18.434 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T02:56:18.434 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-12T02:56:18.434 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 -160us[ -160us] +/- 95ms 2024-03-12T02:56:18.434 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -1410us[-1407us] +/- 65ms 2024-03-12T02:56:18.434 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +1626us[+1626us] +/- 63ms 2024-03-12T02:56:18.434 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T02:56:18.435 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-12T02:56:18.444 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-12T02:56:18.445 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-12T02:56:18.473 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T02:56:18.549 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T02:56:18.624 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T02:56:18.698 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T02:56:18.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:18.730 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T02:56:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:18.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T02:56:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:18.794 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-12T02:56:18.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-12T02:56:18.831 INFO:teuthology.task.internal:Duration was 718.977074 seconds 2024-03-12T02:56:18.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-12T02:56:18.873 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-12T02:56:18.874 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T02:56:18.876 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T02:56:18.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T02:56:18.917 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T02:56:18.922 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T02:56:18.924 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T02:56:19.415 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-12T02:56:19.415 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-03-12T02:56:19.415 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T02:56:19.436 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2024-03-12T02:56:19.436 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T02:56:19.464 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-03-12T02:56:19.465 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T02:56:19.491 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-12T02:56:19.491 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T02:56:19.494 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T02:56:19.507 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T02:56:19.562 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-12T02:56:19.563 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T02:56:19.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T02:56:19.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T02:56:19.751 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-12T02:56:19.762 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-12T02:56:19.762 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T02:56:19.789 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T02:56:19.816 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T02:56:19.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-12T02:56:19.855 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T02:56:19.857 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T02:56:19.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T02:56:19.880 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-03-12T02:56:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2024-03-12T02:56:19.909 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-03-12T02:56:19.975 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T02:56:19.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:19.990 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T02:56:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:20.004 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T02:56:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T02:56:20.031 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-12T02:56:20.042 INFO:teuthology.task.internal:Transferring archived files... 2024-03-12T02:56:20.043 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/remote/smithi079 2024-03-12T02:56:20.043 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T02:56:20.082 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/remote/smithi115 2024-03-12T02:56:20.082 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T02:56:20.117 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827/remote/smithi129 2024-03-12T02:56:20.118 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T02:56:20.153 INFO:teuthology.task.internal:Removing archive directory... 2024-03-12T02:56:20.153 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T02:56:20.155 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T02:56:20.157 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T02:56:20.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-12T02:56:20.214 INFO:teuthology.task.internal:Not uploading archives. 2024-03-12T02:56:20.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-12T02:56:20.330 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-12T02:56:20.331 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T02:56:20.333 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T02:56:20.335 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T02:56:20.345 INFO:teuthology.orchestra.run.smithi079.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 02:56 /home/ubuntu/cephtest 2024-03-12T02:56:20.347 INFO:teuthology.orchestra.run.smithi115.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 02:56 /home/ubuntu/cephtest 2024-03-12T02:56:20.349 INFO:teuthology.orchestra.run.smithi129.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 02:56 /home/ubuntu/cephtest 2024-03-12T02:56:20.350 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-12T02:56:20.362 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-12T02:56:20.443 INFO:teuthology.nuke:Checking targets against current locks 2024-03-12T02:56:20.472 DEBUG:teuthology.nuke:shortname: smithi079 2024-03-12T02:56:20.473 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T02:56:20.497 DEBUG:teuthology.nuke:shortname: smithi115 2024-03-12T02:56:20.498 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T02:56:20.502 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:34:21.062167', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPae1lA8hOXE9C3ebz8WdBAdUiq63/PL99WIrU5rSMdQ/IozNnbs0Is0k8+fjt2DVuUjL9Ucqewi0+ybs21i89M='} 2024-03-12T02:56:20.522 DEBUG:teuthology.nuke:shortname: smithi129 2024-03-12T02:56:20.522 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T02:56:20.527 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:34:21.064070', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG4exXonK73KnghevnJSUvMY6uidkshnKuYn0e5Lr/NDha/kZVTqlucLpvoLCK0BZC+58sIwZBJESSz/LTtV+JI='} 2024-03-12T02:56:20.555 INFO:teuthology.orchestra.console.smithi079:Power off 2024-03-12T02:56:20.556 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T02:56:20.566 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591827', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:34:21.065330', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG1rTH5lOKw+pdfuFit8a68/ZAeNVwpS16sKS0MWrAZRw8TZHmAZlR6foYI+xTa0G9GttzDBCuEDQoVWIn2TN2o='} 2024-03-12T02:56:20.580 DEBUG:teuthology.orchestra.console.smithi079:power off output: Chassis Power Control: Down/Off 2024-03-12T02:56:20.580 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:20.594 INFO:teuthology.orchestra.console.smithi115:Power off 2024-03-12T02:56:20.594 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T02:56:20.607 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-03-12T02:56:20.618 DEBUG:teuthology.orchestra.console.smithi115:power off output: Chassis Power Control: Down/Off 2024-03-12T02:56:20.618 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:20.632 INFO:teuthology.orchestra.console.smithi129:Power off 2024-03-12T02:56:20.632 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T02:56:20.646 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-03-12T02:56:20.659 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2024-03-12T02:56:20.660 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:20.683 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-12T02:56:24.608 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:24.647 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:24.683 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:24.718 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-03-12T02:56:24.757 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-03-12T02:56:24.792 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-12T02:56:28.720 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:28.759 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:28.793 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:28.830 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is off 2024-03-12T02:56:28.869 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-03-12T02:56:28.904 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-12T02:56:28.934 INFO:teuthology.orchestra.console.smithi079:Power off completed 2024-03-12T02:56:32.874 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:32.905 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T02:56:32.984 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2024-03-12T02:56:33.015 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2024-03-12T02:56:33.085 INFO:teuthology.orchestra.console.smithi115:Power off completed 2024-03-12T02:56:33.116 INFO:teuthology.orchestra.console.smithi129:Power off completed 2024-03-12T02:56:33.220 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} duration: 718.9770736694336 failure_reason: '"2024-03-12T02:52:23.500567+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-12T02:56:33.220 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T02:56:33.577 INFO:teuthology.run:FAIL